README.md

Hindi, Hinglish and the English-Trained Robot

A comprehensive study of multilingual understanding, robot execution, and training-free lexical adaptation in Vision-Language-Action models.

I recommend reading the study notebook, it was carefully curated by me for HUMAN reading
All outputs are embedded so a single download of the notebook should work perfectly fine, this README simply restates a portion of the discussion there.
For more information, the supplementary details notebook has more raw details.

Study methodology

Question 1. Identify a critical blind spot, bias, or capability gap in current AI models, drawing from your background and lived experience.

Imagine a robotics company develops a robot in the United States and deploys it in India. An Indian user asks it to perform an action it has already been trained on. Can the user instruct it naturally in Hindi or Hinglish, or must they learn the English terminology familiar to the robot?

As a native Hindi speaker, I find this question personal as literature documenting Hindi in VLA robots is extremely limited. Also, Hindi-English mixing, commonly called Hinglish, is incredibly common in everyday communication, while script choice adds another dimension to how people choose to naturally express themselves.

My research interests lie in computer vision and robot learning, particularly in developing intelligent agents capable of generalizing their acquired knowledge to unfamiliar tasks and environments. This led me to investigate whether the multilingual capabilities of pretrained vision-language models translate into reliable robot execution.

The gap we chose to investigate is native Hindi and Roman Hinglish instruction following in Vision-Language-Action (VLA) models.

Major robot-learning benchmarks, including LIBERO, CALVIN, and RoboCasa, primarily evaluate whether a robot can successfully execute its assigned tasks using English instructions. More recent investigations, such as LIBERO-Plus and LIBERO-Para, examine sensitivity to instruction wording and paraphrasing. Multilingual VLA research has also demonstrated substantial language-dependent execution differences, but Hindi and naturally expressed Roman Hinglish remain relatively underexplored in these evaluations.

Another concern emerged from how multilingual instructions are constructed. Many evaluation settings depend on translations of original English robot instructions. However, linguistically acceptable translations do not necessarily preserve the intended physical objects, actions, destinations, or spatial relationships.

In our translation audit, we observed cases where machine translation changed task-relevant terminology. This raised questions about whether evaluations always distinguish a robot's inability to understand a language from ambiguity introduced through translation.

We therefore investigated three connected questions:

1. What multilingual capability is present in the original pretrained VLM before robot adaptation?
2. How do language and task terminology affect observed robot execution?
3. Could we improve a frozen policy's multilingual performance without expensive retraining?

The broader problem is that a robot may already know how to perform a task, yet fail to execute it reliably when the user expresses the same intention in their native language.

Question 2. Systematically evaluate the gap using a frontier open-weight Hugging Face model between 0.6B and 6B parameters.

Model Selection

We selected the original Qwen3-VL-4B-Instruct and its robot-adapted counterpart, StarVLA/Qwen3-VL-OFT-LIBERO-4in1.

The Qwen3-VL backbone contains ~4.4B params, while the OFT action head adds ~65M parameters, keeping the evaluated architecture <5B parameters.

We chose this model for three reasons.

First, it sits within the requested 0.6–6B parameter range and is a practical model size for repeated evaluation and robot rollouts.

Second, the original backbone has documented multilingual capabilities, allowing us to investigate whether these capabilities remain useful for downstream robot execution.

Third, the adapted checkpoint was trained on LIBERO-All, which includes LIBERO-Goal, a suite of ten manipulation tasks. This allowed us to investigate language sensitivity while holding the robot's environment, assigned task, and trained policy constant.

All evaluations were performed locally using an NVIDIA RTX 3060 with 12 GB VRAM, and not online resources due to the nature of study requiring long hours of consistent and repeated experimental work.

Lastly, it is because the particular OFT action head is simple in design and easy to work with, while also showing significant multilingual capability loss, which we examined during our initial pre-model selection phase.

Our study takes place in multiple phases via multiple experiments.

Experiment 1: Multilingual Understanding in the Original VLM

Before evaluating robot execution, we wanted to establish how well the original Qwen3-VL-4B-Instruct understood Hindi and Roman Hinglish.

For this, we selected 98 matched English-Hindi image-question pairs from IndicVisionBench, an established benchmark for culturally grounded Indian visual-language understanding.

Roman Hinglish was not part of its original evaluation. We therefore selected a subset of the benchmark and constructed Hinglish counterparts while preserving the questions, images, and intended answers.

Results:

The relatively small difference demonstrated substantial multilingual and culturally grounded visual understanding in the pretrained model. Individual questions also showed contrasting outcomes, with Hindi and Hinglish occasionally outperforming English.

We then performed LIBERO-specific comprehension tests to investigate instruction interpretation and visual scene understanding separately.

The original model demonstrated strong understanding of LIBERO task instructions across all three languages. However, interpreting the simulated robot scenes proved more challenging, particularly under Hindi and Hinglish questions.

Experiment 2: Multilingual Robot Execution

Understanding an instruction does not necessarily mean being able to execute it.

We evaluated the frozen StarVLA policy on ten LIBERO-Goal manipulation tasks using their canonical English instructions and corresponding Hindi and Roman Hinglish instructions.

Each condition covered the same 20 initial states per task, giving us 200 episodes per language and 600 matched language-condition outcomes.

The environment, initial states, native task-success criteria, and trained policy remained unchanged. Instruction language was varied during inference, while keeping everything else consistent.

Our results were:

The differences were substantial and varied considerably across tasks.

All three languages achieved perfect success on Bowl on plate, whereas neither Hindi nor Hinglish succeeded on Open middle drawer. Conversely, Roman Hinglish achieved 20/20 successes on Cream cheese in bowl, compared with only 4/20 in Hindi.

These findings demonstrated that the original VLM's strong multilingual comprehension did not translate consistently into successful execution by its adapted robot policy.

The difference between Hindi and Roman Hinglish was particularly interesting. Since both expressed largely equivalent manipulation goals, we hypothesized that familiar English terminology present within Hinglish might be contributing to better execution.

Experiment 3: Robot Behavior and Instruction Sensitivity

While reviewing individual rollouts, we noticed that some unsuccessful tasks appeared to produce very similar robot behaviors even when the instructions requested completely different actions or destinations.

This raised doubts about the policy's sensitivity to task meaning, familiar keywords, instruction structure, and previously learned action patterns.

We investigated these observations through recorded trajectories and 20 exploratory Hinglish and French-English instruction-structure episodes, including experiments like keyword-only prompts, reordered instructions, and negative constraints.

These behavioral inspections definitely cemented the doubt of a default trajectory and also led to finding of an interesting phenomena, of an apparently failed starting trajectory turning to successful via course correction.

Experiment 4: French Lexical Interventions

Returning to the Hindi-Hinglish execution gap, we wanted to investigate whether the presence of familiar English vocabulary influenced multilingual robot execution and eventual high score in Hinglish.

We chose French for additional exploratory experiments because it uses the Latin script while differing substantially from English in its task vocabulary.

Rather than translating complete instructions into English, we experimented with replacing individual object names, action expressions, and destination phrases with English alternatives while preserving the intended manipulation goal.

One particularly interesting comparison involved placing a bowl on top of a cabinet:

Instruction condition
Original French
Replace destination noun with the cabinet
Replace destination phrase with on top of the cabinet

Other recorded comparisons showed that adapting the middle-drawer phrase, introducing the plate into a plate-pushing instruction, or changing le bol to the bowl could alter execution outcomes.

Across these exploratory experiments, we observed that selectively replacing task-relevant expressions could change execution without retraining the robot policy.

The results suggested that the location and degree of lexical adaptation may matter, rather than simply inserting English words indiscriminately.

Nevertheless, the findings provided the motivation for our proposed training-free adaptation approach.

Question 3. Propose a path forward that might close the gap.

Proposed Solution: Scene-Grounded Minimal Lexical Adaptation

Our experiments suggest a simple and practical direction is possible for improving multilingual robot instruction following without modifying the underlying VLA policy.

The central idea is actually simple:

A robot should not need the user's entire instruction translated into English when only one unfamiliar object name or spatial expression may be preventing successful execution.

This is important because our translation audits when evaluating existing benchmarks of multilingual behaviors in VLA showed little focus on translation quality itself. The translation audits showed several popular translater tools changing intended action, which may or may not have affected the existing literature and their results.

But the idea also resembles a common problem in multicultural human communication.

When two people from different linguistic backgrounds use different names for the same physical object, the misunderstanding can often be resolved by identifying the object in their shared surroundings and using a familiar alternative name, without changing the rest of the conversation.

Its not necessary that explicit guidance must be given, quite different, a simple dictionary lookup into synonyms of the word in target language drives the point across.

We propose a scene-grounded, inference-time lexical adapter that applies this principle to multilingual VLA instruction following.

Proposed Mechanism

The adapter would operate between the user's natural-language instruction and the existing VLA, using the original instruction and available camera observations.

Step 1: Understand the requested goal.

A multilingual language or vision-language model interprets the original instruction to identify the requested action, manipulated object, destination, and spatial relationship.

The original language remains the primary source of intention, rather than treating an automatically generated English translation as ground truth.

Step 2: Ground the relevant objects in the scene.

The adapter associates mentioned objects and spatial expressions with visible scene referents.

For instance, it identifies which cabinet, drawer, or bowl the user is referring to before selecting an alternative expression or even a better suited synonym for the object in question.

This is important because a translation may be linguistically acceptable without being sufficiently specific to the physical objects in the robot's environment.

Step 3: Apply the minimum necessary lexical change.

The adapter selects equivalent expressions from a small vocabulary of terms familiar to the downstream robot policy.

Adaptation would proceed from less intrusive to more expressive substitutions:

• Preserve the original instruction whenever no adaptation is needed.
• Prefer replacing a task-relevant object or destination noun.
• Expand to a complete spatial or action phrase only when a smaller substitution is insufficient.

For example:

Original French: Mettez le bol sur l'armoire.

Minimal substitution: Mettez le bol sur the cabinet.

Expanded destination phrase: Mettez le bol on top of the cabinet.

Both alternatives preserve the intended bowl-on-cabinet manipulation while exposing different amounts of familiar vocabulary to the robot.

Step 4: Preserve intention before execution.

Before passing the adapted instruction to the VLA, the adapter checks whether the manipulated object, action, destination, and spatial relationship remain consistent with the original request.

If a substitution introduces ambiguity or changes the physical task, the adapter rejects it.

The existing VLA then receives the adapted instruction alongside its usual camera observations, without changes to its trained weights or action-generation mechanism.

---

Thank You

Download and open study.ipynb locally in JupyterLab, or in Colab/Kaggle.

Download the accompanying data/, figures/, media/, and optionally notebook_backend/ directories as well so that analysis cells and recording links work.

Model and Study Resources

Matched trilingual VLM evaluation: IndicVisionBench English-Hindi-Hinglish evaluation, containing the 98 matched source-derived image questions and recorded predictions. LIBERO-specific understanding items are separate subsets in the rollout repository.

Complete rollout dataset: Multilingual VLA Rollouts, with 1,041 original recordings. The viewer separates task-scene rollouts from six behavioral controls without success labels; exploratory and excluded-pilot cohorts are explicitly identified.

Original pretrained model: Qwen3-VL-4B-Instruct

Adapted robot policy: StarVLA/Qwen3-VL-OFT-LIBERO-4in1

Included pretrained weights: Qwen shard 1 · Qwen shard 2.

Included adapted weights: StarVLA step-50,000 checkpoint, with its original configuration, normalization statistics, and model card.

Robot benchmark: LIBERO

Multilingual visual benchmark: IndicVisionBench

IndicVisionBench is a published dataset; this study uses a selected subset.

Reproducibility scope

Both notebooks run their existing analysis and visualization cells using the included curated records. This recomputes multiple-choice correctness, matched task-success summaries, paired comparisons and bootstrap intervals, and regenerates programmatic figures and diagrams. The loader verifies the retained numerical totals and task/state pairing.

Recorded models:

• Qwen/Qwen3-VL-4B-Instruct.
• StarVLA/Qwen3-VL-OFT-LIBERO-4in1, checkpoint checkpoints/steps_50000_pytorch_model.pt.

The model weights and loading metadata are included under models/. GPU inference software and simulator software are not included.

Repeating model inference requires the appropriate external runtime; repeating robot evaluations additionally requires StarVLA/LIBERO.