00:00:12.559
little bit uh still i don't quite sound like myself right now but uh anyway welcome thank you for
00:00:19.199
coming um so i'm talking about uh mixed reality simulation uh
00:00:25.119
and i guess sort of more the robotics context of that and sort of the high level overview as to
00:00:30.800
why we might want to do something like this and how ruby ties into it um
00:00:35.840
first of all how many of you here are robotics people anyone do any robotics
00:00:42.399
cool all right uh a couple of people sounds good uh and then anyone
00:00:48.079
uh know about dragon ruby or have played with it okay so cool there's a good assortment
00:00:54.640
there um okay so i'm coda weaver uh i'm a founder of a
00:01:01.120
robotics company out of boston uh called sky technologies we basically try to
00:01:06.640
build robots that basically are human aware so they navigate without
00:01:13.600
running into people and without slowing down much and things like that and also we do robot arm
00:01:19.119
manipulation and things like that so this is a robot that we have uh out on the market that we collaborated with uh
00:01:25.119
dmg modi which is a machine tools company and you know we build
00:01:30.960
software for what's like that um i primarily work with c plus plus and python so admittedly ruby's not actually
00:01:38.960
my uh my bread and butter um our stack also has a lot of elixir and
00:01:44.159
rust um so simulators most of our work
00:01:49.439
is in simulators right so we start off by building a model putting it in a simulation so this is
00:01:55.920
gazebo here um which is used all over the place it's the standard one for ross
00:02:01.360
and um nasa uses it for their their robots as well and so this right
00:02:08.800
here is an omnidirectional wheel omni wheel and so we actually have each of the barrels
00:02:14.879
simulated as well and simulators are a lot of fun to play with so this is basically four omni
00:02:21.280
wheels on a vehicle that's kind of rolling down a hill um and
00:02:27.360
you know you can see that i think the video should show uh
00:02:33.519
all of the individual barrels and things like that rotating so you can actually see you can do pretty comprehensive
00:02:38.879
simulations just on uh pretty you know standard hardware
00:02:44.560
uh now there is this one problem where when we look at this
00:02:50.720
and there's a robot here we actually don't really know
00:02:55.840
how fast it's driving or how big the robots are when you just look at them right
00:03:01.120
um so that's actually a pretty common problem where you start building a lot of things in simulation and spend too
00:03:06.959
much time there and if you don't really have that feedback loop with a real robot
00:03:12.159
you kind of lose sense of reality so that's that's actually a really key point where
00:03:18.400
so this robot was driving about 0.5 meters a second which you know is actually pretty slow
00:03:24.239
but it's really hard to tell just from that so that's that's a standard issue that we end up running into pretty often
00:03:30.720
especially when we try to present things to customers in addition when you actually have these
00:03:36.080
robots in various environments that that impacts as well because in a very large space you know robot looks
00:03:43.599
much slower than it than it uh that might actually be moving so all of that really plays a big part of
00:03:50.959
of the kinds of work that we end up doing so sometimes simulations also do some
00:03:58.159
kind of funky things so you'll notice that this one here kind of flipped over
00:04:05.439
not actually sure why that did that but they're very often pretty finicky in terms of the parameters and depending
00:04:12.640
on the uh integrators and things like that you end up running into issues like that
00:04:18.560
um here's another case where it's very unstable and
00:04:24.160
uh don't worry that's that's not the robot that's sent to customers um
00:04:29.759
you know they they do drive a little bit better than that um but you know these are typical issues
00:04:35.919
that we run into so you know we began looking into better options for simulators and what
00:04:42.880
we wanted were things that were you know not easy to destabilize we didn't really need anything that was very complex
00:04:49.040
right because in vast majority of our cases we're dealing with vehicular uh you know
00:04:54.639
basically 2d simulations because these are just on planes and then for like robot arms and things
00:05:00.639
like that that's not necessarily something we need to handle uh in in this case uh headless is
00:05:07.360
important so that we can do things in ci uh fast startup you know this one of those things where
00:05:13.280
the harder it is to actually boot something up where the longer things take the slower the development cycle so
00:05:18.960
that ends up impacting us quite a bit and again 2d is it's fine by me
00:05:25.199
and another key thing is it's easy to describe various scenarios and environments and we couldn't really find anything for
00:05:32.320
our needs here so um i didn't really find anything yet so
00:05:37.680
we're working on something for that but uh you know that that's something which is a typical problem that we run into
00:05:44.080
and a lot of people robotics uh do as well so
00:05:49.360
i ran into dragon ruby while looking for a simple game engine to start building a
00:05:55.120
simulator that better fit my needs uh so dragon ruby is a
00:06:02.160
pure code game engine so there's no unnecessary ui i don't know how many of you have worked with other game engines
00:06:08.479
but very often there's like these really complicated uis that just you know make it a lot harder to just get started
00:06:15.440
um as someone who primarily builds robots i want to keep that learning curve as low as possible because i don't
00:06:21.520
spend that much time doing that um so that's the second piece it's very easy to get up and running which means
00:06:28.240
the other members of my team who uh you know spend all their time in elixir
00:06:33.680
or in python or in c plus plus you know they don't have any problems
00:06:38.960
getting started with this either the api also is very very small
00:06:44.479
and it's used to make real video games which amir who's who's sitting over there
00:06:50.800
who's the founder of dragon ruby uh used to develop games for mobile and for various console
00:06:57.759
platforms as well so i did promise you that it's pretty
00:07:03.039
easy uh so i figured i would do a little bit of live coding um how is everyone's high school physics
00:07:11.440
okay all right good okay um we'll see if mine is okay too but uh
00:07:16.960
let me see okay so i'm just going to bring up
00:07:24.560
some code okay so the very first thing i'll do is uh right now i am in
00:07:31.280
none of you can see that uh okay so right now i am in just the dragon ruby's
00:07:38.560
root directory so in order to create a new project and amir is not gonna like that i'm not using his method but i'm
00:07:45.360
just going to create a new directory and i'll just call it springs i guess
00:07:51.199
and we'll just add a folder called app
00:07:59.039
we go and in here i'm just creating a file called main.rb
00:08:04.960
so this is um just the you know main file that
00:08:11.360
will be working out of here okay so now inside of this okay you can see that
00:08:16.560
good uh we can create the main loop this is the main event loop and now
00:08:23.360
if we go into the terminal and there are my uh let's see springs
00:08:30.400
and springs
00:08:35.440
there we go okay so then if i go up one and call dragon ruby
00:08:40.800
on this directory you will see that we now have our dragon ruby window so
00:08:48.399
one of the cool things about dragon ruby is that we can actually just start typing away and whenever we save it hot
00:08:54.480
loads so uh if we for example create
00:09:00.160
a rectangle uh or square i'll just put this in the middle of the of the screen
00:09:13.279
and height of 40 and i hit save and now we have a box in the middle of our screen it's
00:09:20.160
very very easy um now we also actually want to move this thing so maybe the first thing
00:09:25.200
we'll do is um we'll use a variable instead
00:09:31.440
so in dragon ruby we use the state which basically helps to serialize and deserialize things um
00:09:38.560
so we can save and load uh data much more easily um and so this kind of where we store a
00:09:44.880
global state as well uh okay so then you know we'll just use
00:09:57.040
if ours dot state dot tick count so basically when we start up
00:10:02.640
we want to call this um and we can also have it reset upon saving with the thing
00:10:08.560
on the top then uh you know we also are going to do
00:10:14.959
our initial velocity um so of course and then we can take
00:10:22.640
our x from there so now if i go ahead and change this to a different value for
00:10:29.040
example you know 100 it'll shift to the left right
00:10:34.480
and so let's see yep um okay so now we want to have
00:10:42.000
this move as a spring right so the basic equations
00:10:47.040
that we need for this are f equals negative k times x uh which is the spring equation so k is
00:10:53.920
spring constant x is a displacement spring right so this pretty simple
00:10:59.680
function uh or yeah equation uh and so we'll just use
00:11:06.160
i'm cheating a little bit here because i know i tested this out earlier uh values that
00:11:13.279
work uh so for dx i'm gonna use the center of the screen here so that's 1280
00:11:18.800
over two uh and then we're looking at the displacement so our rx.x
00:11:24.880
minus that is our rtx here and then you know because dragon ruby
00:11:31.279
works at a fixed tick time rate so it's 60 frames a second you know we could have a dt in here but
00:11:38.240
we can also just assume that it's tied into the parameters um so
00:11:44.000
the next one that we do is f equals ma of course uh the famous one right
00:11:50.880
um so we can just assume a mass of 10 units um and then
00:11:57.760
for this v equals a times t uh so velocity and in this case because
00:12:05.360
we're assuming it's some constant we're not dealing with real world units right now
00:12:10.800
uh and so therefore all we need to do is add
00:12:17.360
let's see vx plus equals v or vx yep okay
00:12:23.040
um so this is basically where every time we're adding to our current velocity the
00:12:28.880
acceleration um and then finally we can get our velocity by integrating
00:12:35.040
the or we can get our position by integrating the velocity so it's
00:12:40.959
just oh yep
00:12:55.440
yep okay cool um
00:13:00.519
ours.x uh plus equals and then the current velocity which
00:13:06.399
and i believe if i save now we should get no movement because it's still in the
00:13:12.399
center but if we instead move this so recall that
00:13:18.000
just by doing something like this we can shift it to the left
00:13:23.920
and then um oh did something here i guess oh uh
00:13:32.160
that's what i did wrong okay uh
00:13:37.200
x equals 100 and then it should shift to the left
00:13:42.639
shouldn't move but it's moving okay well uh
00:13:49.600
point being uh it's supposed to do something like
00:13:58.639
okay okay that one's broken too i guess um
00:14:04.160
anyway point being with just a few lines of code you can actually start creating simulations
00:14:09.279
uh in a very kind of intuitive way um so
00:14:15.120
dragon ruby basically this was the enabling technology for
00:14:21.040
a lot of the kinds of things that we're looking for um so i very quickly got addicted
00:14:28.160
uh and on the first day of dragon ruby this is what i made um so it's a little you know
00:14:35.519
system to create various shapes i guess it's almost like a
00:14:41.120
yeah vector editing tool you can save and load images and things like that
00:14:46.240
um second day was this little physics simulation a little soft body kind of
00:14:52.079
thing so you have you know these spring spring-like systems and various like cloth like
00:14:58.399
um things and on you know the next week
00:15:04.000
i was doing some 3d stuff note that this is a 2d engine and then the next day i was doing i know
00:15:11.519
some of you were here for the drone talk earlier uh this is 2d uh two
00:15:17.519
2d on uh or 2d game engine drone game i guess of sorts and then you know
00:15:24.320
within a month of using dragon rubios doing things like this which is like a little 3d modeling tool
00:15:30.639
using dragon ruby now the main point of all this is that
00:15:35.920
dragon ruby's incredibly fun very very quick simple and you know it solves a
00:15:41.600
lot of the problems that i was trying to solve in in my robotics problems right
00:15:48.240
um so uh the next thing projectors
00:15:54.480
uh so the issue that we're running into with
00:16:01.519
uh simulations largely of just not knowing what the right scale is things like that
00:16:07.279
um you know you naturally go towards using real robots as well
00:16:13.040
right you know that's very important part of the development cycle and so when we run the real robot
00:16:20.160
well there are some problems there where it's still difficult to actually visualize what you're doing
00:16:26.160
and in your physical space there's a pretty limited uh number of options as to what you can
00:16:32.959
do our office is uh we have about a 3000 square foot space for robot testing
00:16:40.160
which um maybe is about oh let's see about a third of this room probably in
00:16:46.720
total uh or yeah maybe a little bit more than that but it's it's not a huge amount of space and you know as a little startup
00:16:53.360
that's the kind of issue that we run into and even when working with big companies they usually don't let us just
00:16:59.360
drive everywhere it's usually a pretty localized space um so but at customer sites we actually
00:17:05.280
very often deal with having to drive two or three hundred meters uh sometimes
00:17:10.319
even a kilometer from one side to the other factory um and in addition the configuration is
00:17:15.760
always very different right so um you know what we do with the real robots we
00:17:21.199
tune the parameters we get a better sense of the real motion and we deal with situations that also aren't hard to
00:17:27.280
simulate which is very often human behaviors and various sensor sensor information so
00:17:33.919
there's actually kind of issues on both sides here um so in order to solve that i um i
00:17:41.360
bought projectors i bought probably about 15 projectors
00:17:46.799
and i tried to put them on the ceiling um so
00:17:52.559
this was this was my first try uh didn't didn't really go so well um
00:17:58.720
you know a couple other angles of that of that same attempt uh you know turns out you
00:18:04.480
if you try to put a piece of wood that's bigger than the uh the opening in the ceiling it's not gonna fit
00:18:10.559
uh but eventually after some maneuvering i managed to get it um i had to take it
00:18:15.840
apart assemble it in in the ceiling as well so that was that was a little bit tricky um
00:18:21.520
but the point being here uh what we're doing is lining the ceiling with these projectors
00:18:27.120
and what that lets us do is it allows us to merge you probably can't read that in the back
00:18:33.200
um so essentially what we're doing is we're taking data from the robot sensor data
00:18:38.320
from the robot and we're putting it into a little simple simple simulation
00:18:43.840
and we merge the lidar data or the the sensor data and send it back to the robot and the robot then operates on
00:18:51.440
that then uh from there we also send that data to the projectors and we send it to the
00:18:58.960
host who might be manipulating the simulation in some way
00:19:04.400
so this is actually you know it's comprised of a few different languages a few different systems uh so the robot
00:19:11.120
primarily uh elixir c plus plus python and um
00:19:17.919
there's some rust and we actually do use ruby and slim
00:19:23.360
for basically describing the configuration of the robot and then
00:19:30.400
we send that to the simulation which is in ruby and then from there we actually use zmq
00:19:39.200
to pass this data to dragon ruby in part because dragon ruby doesn't natively have tcp
00:19:46.160
support yet so there's this nice c extension feature
00:19:51.440
where you can write additional parts and see so
00:19:56.720
i added in some zm2 support for that and then we tied in
00:20:02.799
uh the all the components that way and so
00:20:08.559
this is sort of view of setting setting everything up so uh you can see
00:20:14.080
that's actually the the black box is there that was about uh probably about three minutes of or five lines of dragon
00:20:21.440
ruby code to get that to visualize um so really kind of the big thing here is
00:20:26.960
that dragon ruby has really really helped us speed up this process because if i was just even if i was to go ahead
00:20:32.960
and grab an image from the internet of a grid and plop it on
00:20:38.480
there it takes a lot more time it takes time to calibrate these things make sure everything's in the right dimensions
00:20:44.400
um so for us this has been a really really easy solution
00:20:49.760
and because of the hot loading i can also actually just directly uh
00:20:55.360
do that on the projectors themselves so if if i need something very quickly i just sh into that and then whenever i
00:21:02.159
save it automatically reloads what's showing on the projector so there's also this client here
00:21:11.919
which this is what we actually use to manipulate the environment so we can zoom in we can rotate we can move things
00:21:19.120
around uh this blue section here um that's a uh that's a dynamic obstacle so to
00:21:26.720
speak so basically it moves between the two points and then the green is the location of
00:21:31.840
the robot and this client is actually the exact same thing that we use to actually to project down onto the floor
00:21:38.640
as well so kind of the big point here is yeah so
00:21:45.120
it's all written in dragon ruby you can manipulate everything and also everything oh
00:21:51.360
important point all this actually runs directly in raspberry pi's as well so you can keep things very very low cost um
00:21:59.039
so kind of putting it all together you end up with uh something kind of like this
00:22:04.640
where um you know we have a robot you can see that i need probably brighter projectors
00:22:12.799
uh and that you know we actually have things pretty well calibrated here um
00:22:19.840
you know it this is actually feedback from from the uh from the simulator to the robot so
00:22:27.520
you could see the robot slowed down a little bit when it went towards the blue box so it waited for the for the obstacle to pass by
00:22:35.039
so that's basically um you know the kinds of things that we've been working on with
00:22:40.400
ruby and simulation um as far as kind of the future plans that we have uh
00:22:45.919
there's a lot of sensor data that i want to visualize um some of that i'm not sure dragon ruby is
00:22:51.679
quite fast enough for yet but maybe i need to finagle that a little bit and get it to go
00:22:58.720
to be more performant i guess on my own code yeah but um and then i also want to add more
00:23:04.320
projectors so this thing tends to scale pretty well so instead of the
00:23:09.360
uh six that we have mounted right now i want to scale it up so that the entire space is covered
00:23:15.200
and also uh we do have hardware-in-the-loop simulation for robots as well so figure that's kind of
00:23:21.360
the next thing to tie in we can have multiple robots in this sort of
00:23:26.960
mixed reality space where we have virtual robots interacting with real robots and
00:23:32.320
hoping to actually get that done for today but you know sometimes doesn't work quite that way um
00:23:38.720
anyway i think that uh wraps up what i had to talk about um so
00:23:46.000
i don't know if there are any questions they can thank you
00:24:04.480
sorry uh the question was uh so as far as the visualization you're able to get that running inside of the
00:24:10.080
projector no no no i have a raspberry pi connected to the projector
00:24:15.679
yep up in the ceiling okay makes sense for certain definitions of
00:24:20.799
sense yeah i i think you probably are aware that uh dragon ruby probably does not run on
00:24:27.840
projector hardware yet so yeah
00:24:33.039
how do you handle the dynamic like obstacle stuff so you don't actually have obstacles in the room it was just
00:24:41.039
an area yep yeah so uh essentially a vast majority of
00:24:46.400
this kind of data that we deal with basically you know we have cameras we have lidar sensors
00:24:53.039
uh so for the lidar basically that's just 2d uh
00:24:58.640
ray it's just a ray 1d ray and they rotate this around very quickly
00:25:03.840
uh so we get these very very fast scans we get i think 40 scans a second
00:25:09.600
or 40 of these full scans a second and each of those consists of 1080 scans so you get
00:25:16.240
um you know this very responsive thing and so we actually send that data up
00:25:21.679
and then we do basically a min on that versus what the simulated robot
00:25:27.039
would be seeing so we basically project from that what it would what it would look at um so it's yeah and then we send that
00:25:34.799
result back to the robot uh as far as like camera data that becomes a lot trickier um
00:25:41.279
you know i think one of the approaches that we're just that we we've also taken here uh is
00:25:47.200
a lot of this stuff just basically gets turned into a higher level a more abstract
00:25:53.360
characterization through the form of cost maps so this is basically like an image of
00:25:58.559
what's what's occupied and what's not um what's dangerous and what's not so we also do
00:26:04.799
uh convert to that kind of thing and send it back there and then yeah
00:26:11.039
yeah
00:26:17.039
anything else
00:26:34.880
uh pie threes doesn't really work that well yeah you have to go with it by four yeah
00:26:40.880
yeah it's actually uh pretty surprising just how big a gap there is between the two
00:26:46.480
when it comes to running this stuff so yeah um and then actually admittedly more
00:26:53.039
recently we uh we've purchased uh ibcs basically for
00:26:59.760
to to replace them something a little bit more powerful
00:27:23.600
uh so i think the question was basically to do the rendering on the raspberry pi or unlike whatever server it is and basically just send out
00:27:30.720
the result to two other machines right uh so i thought about that um part of
00:27:36.159
that is just i think there's a scalability issue uh when it comes to that so
00:27:41.200
when we're talking about having 10 projectors or whatever it is it that becomes a lot more difficult
00:27:47.360
so so that's part of why we didn't do that
00:27:52.480
yep
00:28:06.000
right yeah so um you know now we're talking about like real robotics questions right
00:28:12.480
so um what we typically do uh i think most popular ways are using what's called a common flower
00:28:19.039
i guess monte carlo based like localization so particle filters various filtering
00:28:25.600
techniques where essentially you have a map a preset map and um
00:28:31.200
you know the robot looks at its sensors and does some sort of calculation where it says oh what's the most likely place
00:28:37.600
for me to be and basically continually updates um so if you're interested um
00:28:43.679
like bayesian filtering techniques and things like that are are the thing to look at
00:28:49.440
there's a lot there
00:29:01.440
yep yeah so there's there are a lot of different localization technologies um
00:29:07.039
ultimately that it usually helps to combine them in various ways so if you can merge them
00:29:13.679
you know yeah it usually helps
00:29:26.080
you don't want to invest a 300 000 robot okay um
00:29:32.240
uh yeah i mean you know there are a few things on sparkfun that are
00:29:37.600
you know usable the problem with these things is that a lot of these hobby grade things just
00:29:43.440
either aren't powerful enough to be able to even run a lot of the kinds of things that you'd want to or but
00:29:49.440
that's something that we can definitely talk offline about if you're interested in like assembling some sort of kit for
00:29:55.840
that you know but
00:30:07.600
so um we used to do all our own hardware uh then we realized
00:30:14.000
that we're too small a company to be able to do that uh and there's a lot of q a or there's basically a lot of
00:30:19.840
maintenance related things that we don't want to do so we basically partner with very you know with larger companies and
00:30:25.200
provide the software for that okay
00:30:32.480
yes i think that's that's it right we're yep okay so thank you very much