Object Interaction

These actions allow the agent to interact with Sim Objects in the scene in various ways.

Pickup Object

Pick up an Interactable object specified by objectId and move it to the Agent’s Hand. Compatible objects have pickupable return true in their Object Metadata and return their current state with isPickedUp. Note that the agent’s hand must be clear of obstruction or this action will fail. If the target object being in the Agent’s Hand would cause it to clip into the environment, this will fail.

Picked up objects can also obstruct the Agent’s view of the environment since the Agent’s hand is always in camera view, so know that picking up larger objects will obstruct the field of vision.

Moveable Receptacles: Note that certain objects are Receptacles that can themselves be picked up. If a moveable receptacle is picked up while other Sim Objects are inside of it, the contained objects will be picked up with the moveable receptacle. This allows for sequences like “Place Egg on Plate -> Pick Up Plate” to move both the Plate and Egg.

event = controller.step(dict(action='PickupObject', objectId="Mug|0.25|-0.27"))

Put Object

Attempt to Put an object the Agent is holding onto/in the target Receptacle. Valid target receptacle objects will return receptacle as true in their Object Metadata.

event = controller.step(dict(action='PutObject', objectId = "Tomato|0.1|3.2|0.43", receptacleObjectId = "TableTop|0.25|-0.27|0.95"))

Drop Object

Drop an object currently in the Agent’s hand and let Physics resolve where it lands. Note that this is different from the Put Object function, as this does not guarantee the held object will be put into a specified receptacle. This is meant to be used in tandem with the Move/Rotate Hand functions to maneuver a held object to a target area, and the let it drop.

Additionally, this Drop action will fail if the held object is not clear from all collisions. Most importantly, the Agent’s collision will prevent Drop, as dropping an object if it is “inside” the agent will lead to unintended behavior.

event = controller.step(dict(action='DropHandObject')))

Throw Object

An extention of the Drop function-throw an object currently in the Agent’s hand in the current forward direction of the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to throw the same distance.

event = controller.step(dict(action='ThrowObject', moveMagnitude= 150.0 )))

Push Object

Push an object specified by objectId away from the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. Only certain objects can be pushed (ie: Chair, Ottoman, Apple, CellPhone etc.). Some objects are structural and cannot be pushed (ie:Refrigerator, Countertop, Microwave). The target object cannot be in the Agent’s hand.

event = controller.step(dict(action='PushObject', moveMagnitude= 150.0 )))

Pull Object

Pull an object specified by objectId towards the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. Only certain objects can be pushed (ie: Chair, Ottoman, Apple, CellPhone etc.). Some objects are structural and cannot be pushed (ie:Refrigerator, Countertop, Microwave). The target object cannot be in the Agent’s hand.

event = controller.step(dict(action='PullObject', moveMagnitude= 150.0 )))

OpenObject

Open an object specified by objectId. Compatible objects will return openable as true in their Object Metadata, and return their current open state with isOpen.

The target object must be within range of the Agent and Interactable in order for this action to succeed. An object can fail to open if it hits another object as it is opening. In this case the action will fail and the target object will reset to the position it was last in.

event = controller.step(dict(action='OpenObject', objectId="Fridge|0.25|0.75"))

Here is an example of opening the Fridge halfway:

event = controller.step(dict(action='OpenObject', objectId="Fridge|0.25|0.75", moveMagnitude = 0.5))

CloseObject

Close an object specified by objectId. Compatible objects will return openable as true in their Object Metadata, and return their current open state with isOpen.

The target object must be within range of the Agent and Interactable in order for this action to succeed. An object can fail to open if it hits another object as it is closing. In this case the action will fail and the target object will reset to the position it was last in.

event = controller.step(dict(action='CloseObject', objectId="Fridge|0.25|0.75"))

Toggle On

Toggles an object specified by objectId into the On state. Compatible objects will return toggleable as true in their Object Metadata, and will return their current on/off state with isToggled. Noteable examples are Lamps, Light Switches, Stove Knobs, and Laptops.

event = controller.step(dict(action='ToggleObjectOn', objectId= "LightSwitch|0.25|-0.27|0.95")))

Toggle Off

Toggles an object specified by objectId into the Off state if applicable. Compatible objects will return toggleable as true in their Object Metadata, and will return their current on/off state with isToggled.

event = controller.step(dict(action='ToggleObjectOff', objectId= "LightSwitch|0.25|-0.27|0.95")))

Slice Object

Slices an object specified by objectId. Compatible objects that have sliceable return true in their Object Metadata. This action destroys the source object and spawns in multiple “sliced” objects based on the source object in the exact same location. This is a one-way state change, so only a scene reset will revert sliced objects to their whole versions. Sliced objects will still report metadata information (isSliced) even after being destroyed. This allows you to check the last position the source object was before the Slice action finished.

event = controller.step(dict(action='SliceObject', objectId= "Potato|0.25|-0.27|0.95")))

Break Object

Breaks an object specified by objectId. Compatible objects have breakable return true in their Object Metadata. This action will cause a visible change in the target object to show it is broken. This is a one-way state change, so only a scene reset will revert sliced objects to their whole versions. Broken objects will still report metadata information (isBroken) even after being destroyed. Some breakable objects shatter completely into pieces, while others will have a visual change such as a cracked screen.

event = controller.step(dict(action='BreakObject', objectId= "Potato|0.25|-0.27|0.95")))

Dirty Object

Makes an object specified by objectId look dirty or messy. Compatible objects have dirtyable return true in their Object Metadata and return their current state with isDirty. This only works on objects that return isDirty = false.

event = controller.step(dict(action='DirtyObject', objectId= "Bed|0.25|-0.27|0.95")))

Clean Object

Makes an object specified by objectId look cleanor tidy. Compatible objects have dirtyable return true in their Object Metadata and return their current state with isDirty. This only works on objects that return isDirty = true.

event = controller.step(dict(action='CleanObject', objectId= "Bed|0.25|-0.27|0.95")))

Fill Object With Liquid

Makes an object specified by objectId filled with liquid specified by fillLiquid. Compatible objects have canFillWithLiquid return true in their Object Metadata and return their current state with isFilledWithLiquid. Only objects that are empty and return isFilledWithLiquid = false can be filled.

event = controller.step(dict(action='FillObjectWithLiquid', objectId= "Mug|0.25|-0.27|0.95", fillLiquid = "coffee")))

Empty Liquid from Object

Empties an object specified by objectId if it is currently filled with liquid (isFilledWithLiquid = true). Note that objects filled with liquid will automatically empty themselves if rotated to an angle extreme enough that the liquid would fall out.

event = controller.step(dict(action='EmptyLiquidFromObject', objectId= "Mug|0.25|-0.27|0.95")))

Use Up Object

Uses up parts of an object specified by objectId. Compatible objects have canBeUsedUp return true in their Object Metadata and return their current state with isUsedUp. This is an action specifically for objects like ToiletPaper, TissueBox, and PaperTowel that all can have bits of themselves used up to cause a change in state.

event = controller.step(dict(action='UseUpObject', objectId= "ToiletPaper|0.25|-0.27|0.95")))

Change Room Temperature Decay Time for Specific Object Type.

This changes all objects specified by objectType in the current scene to have a new Room Temp Decay Time specified by TimeUntilRoomTemp. This can be done to adjust the amount of time it takes for specific object types to return to room temperature. By default, all objects begin at Room Temperature. Objects placed on/in Hot or Cold sources (stove burner, microwave, fridge etc.) will have their ObjectTemperature value changed to Hot or Cold. If the object is removed from a Hot or Cold source, they will return to room temperature over time. The default time it takes for an object to decay to Room Temp is 10 seconds.

event = controller.step(dict(action='SetRoomTempDecayTimeForType', objectType= "Bread", TimeUntilRoomTemp = 20.0)))

Set Global Temperature Decay Time

This changes all objects in the current scene to have a new Room Temp Decay Time specified by TimeUntilRoomTemp. By default, all objects begin at Room Temperature. Objects placed on/in Hot or Cold sources (stove burner, microwave, fridge etc.) will have their ObjectTemperature value changed to Hot or Cold. If the object is removed from a Hot or Cold source, they will return to room temperature over time. The default time it takes for an object to decay to Room Temp is 10 seconds.

event = controller.step(dict(action='SetGlobalRoomTempDecayTime', TimeUntilRoomTemp = 20.0)))

Disable Temperature Decay

This disables the decay over time of the ObjectTemperature of objects. If set to false, objects will not decay to room temperature and will remain Hot/Cold even if removed from the Hot/Cold source.

event = controller.step(dict(action='SetDecayTemperatureBool', allowDecayTemperature = false)))

Next Steps

Continue on to the Held Object Manipulation documentation for information on how to manipulate picked up objects.