Take a team of smart students from the Eindhoven University of Technology (TU/e) with a vision to develop an autonomous drone that adds real value in healthcare and related fields, make them aware of the ever-present threat of children drowning in swimming pools and you’ve got a challenge to get passionate about.
“I may be more experienced, but these young people come with fresh ideas. We learn from each other.”
Despite the best efforts of thousands of well-trained life-saving professionals and volunteers, on average each year 60 people will suffer injuries as a result of almost drowning in Dutch swimming pools, and 2 will actually drown. By far the most common group of victims is children.
According to Anja de Brouwer, a long-serving volunteer at the ‘Eindhovense Reddingsbrigade’ (the life rescue volunteers service in Eindhoven), who is trained to realistically simulate accident victims, “When a child’s drowning they usually don’t shout or wave their arms as you’d expect. It just goes quiet and, before you know it, you’ve lost them.”
Serious roles
Anja is one of the many volunteers who recently took part in a large-scale event at the Pieter van den Hoogenband Swimming Stadium organized by the rescue service to help the Blue Jay Eindhoven team develop a drone that can assist lifeguards by performing three key tasks: monitoring activity in a busy pool, detecting a swimmer in distress and responding appropriately.
During the evening, eight cameras continuously film the volunteers as they simulate drowning people. “To learn how to detect which movements mean someone may be in trouble rather than playing, the drone must be shown lots of different videos of drowning people,” explains Liza Boormans, the Blue Jay team manager. “Once it has enough data, it can recognize and detect drowning behavior itself.” Thankfully, film footage of people really drowning is rare. Hence the need to film the LOTUS volunteers role-playing victims.
Not waving but drowning
This process of detecting someone drowning is one of the real innovations of the lifeguard assistant project, and one of its biggest challenges. “There are already detection methods on the market that spot if someone has drowned, but the Blue Jay team is developing a drone to detect if someone is about to drown — in time to help them. That’s a much more complex challenge,” says Arjen van de Wetering, an Internet of Things (IoT) consultant at IBM who has been supporting the Blue Jay team since IBM became a main partner to the project a few months back.
“Distinguishing between play and distress is difficult using traditional analytics methods” explains Arjen. “So the team is using artificial intelligence (AI), employing IBM’s Watson technology. The team will use the same drowning simulation videos to train the system and to test whether it can make the drowning-playing distinction, and later whether it can make that judgement effectively in real-life situations.”
This detection process isn’t simple. Having singled out individuals from its aerial view of the whole pool, the system must follow people using approximately 2 seconds of consecutive video frames to determine how someone is moving. The oldest frame from those 2 seconds is then dropped and replaced by a new one. The system is thus continuously analyzing how people move within a 2-second timeframe. The calculations to determine whether a swimmer’s movements constitute playing or drowning take a huge amount of computer power. In our first pilot setup, all these videos are streamed live from the cameras to the cloud, where the calculations are then done on IBM’s Watson AI mainframes. The ‘decision’ about the swimmer’s movements is then made by Watson and sent back in a matter of milliseconds to the drone, which acts accordingly — a classic IoT solution. For the final system, the team probably don’t want to depend on an unreliable Internet connection, so it might consist of a trained Watson-based application on-premise. But that decision has yet to be taken.
International and multi-disciplined
Blue Jay Eindhoven is an on-going project with the ultimate goal of developing a drone that can assist professionals in healthcare settings. The team is made up of students from the Technical University of Eindhoven (TU/e) who take time out from their studies to join Blue Jay for a year. “Because you get no elective points or financial support for working on Blue Jay,” explains Liza, “only people who are very committed to our goals and vision join the team.” Liza’s own role as team manager ends in September when she returns to her full-time studies in Medical Sciences and Technology.
The current team is an international group of 18 students from 9 different academic disciplines. As the Blue Jay personnel change pretty well entirely each academic year, continuity is assured through an advisory board that includes some of the previous year’s students who support the next year’s team. “The board ensures the team follow the long-term Blue Jay vision,” says Liza, “but gives them sufficient freedom to add their own insights and innovations.”
Last year’s team focused on developing a drone to assist in the Máxima Medical Center near Eindhoven. Much progress was made. It could, for example, play games with child patients. But a stumbling block, which has not yet been overcome, is noise: a drone is still pretty noisy and that obviously isn’t handy in a hospital environment.
The perfect setting
Ironic, then, that it was during a brainstorming session on solving the noise problem that the team heard about the case of a 9-year-old refugee girl who drowned in a swimming pool in Rhenen. They immediately saw the potential for a Blue Jay drone to help prevent such tragedies. “Swimming pools are a perfect setting,” explains Liza, “Noise isn’t an issue, but we can still demonstrate the huge potential value of the drone: the ideal proof of concept.”
But as team project manager, Mike van Sighem, points out, the swimming pool drone has its own challenges. “As well as drowning detection, it must be able to autonomously navigate buildings, notify personnel and bring bulky and heavy objects to an accident scene.”
Pim van Dommelen, Blue Jay’s mechanical engineer, sees this last as probably the biggest mechanical challenge of the lifeguard drone. “It needs to be light, otherwise you have to switch batteries every few minutes and a drone that’s out of the air too often isn’t much use as a lifeguard assistant. But at the same time, it must be powerful enough to fetch heavy objects such a lifebuoy, defibrillator or first aid kit.”
Partners make perfect
NXP, the second main project partner, is helping the team with making the drone both robust and responsive enough to do its job. “Last year we helped the Blue Jay team develop a robust communication channel between ground station and drone,” explains Gino Knubben, Principal System Engineer at NXP and closely involved with the company’s support to Blue Jay. “This year they’re using a slightly newer version of the same technology, but our role is even more drone-focused, looking mainly at two major systems. First, the flight control system, which deals with the drone’s movement, orientation, balance and remote control – the team chose this year to switch to our flight controller, NXPhlite. And secondly, the application processor, which deals with the display and the interaction.”
For the project, functionality is critical and one big benefit of working with world-class companies like NXP, IBM and Fourtress, the third and final main Blue Jay sponsor, is the Blue Jay team’s access to advanced technology not found in consumer products. “Compared to regular Wi-Fi on your phone,” explains Gino, “our automotive-qualified connectivity solutions are superior in robustness, RF reflections, power consumption and the number of ad-hoc connections you can do per second.” The robust connections come from 802.11p technology used in the automotive industry, and can handle the drone’s rapid movement and ad-hoc communications. Gino laughs, “Two years ago, everybody had to remember to switch off the Wi-Fi on their phones before testing or demonstrating. Now with 802.11p we’ve resolved that particular interference issue.”
One-stop shop
Another benefit of having NXP as a partner is that, unlike many semiconductor manufacturers, they can provide a wide range of components. “We’re providing a large part of the electronics they need for drone development,” says Gino. “Not just the wireless link but also high-performance microcontrollers; proven reference boards with all the components assembled; and even complete sub-systems, including supported Operating Systems. The team then adds their peripherals, displays, sensors, applications, graphical user interfaces, etc.”
But Gino is first to say the collaboration is not one-way traffic. “The Blue Jay team are basically beta-testing some of this technology for us and their feedback is invaluable. For example, on development of the FMU (Flight Management Unit): along with all the other partners involved, they provide feedback, they test… It’s outside the students’ comfort zone and the remit of the project. So that wider commitment is great. They also visit us at NXP on a regular basis, which is a lot of fun.”
And there are areas where old hands learn from the students. “They’re very knowledgeable, for example, on the ‘total drone’, such as how to calibrate and configure it. And have some seriously skilled drone pilots! Another example is hardware assembly experience. We’d been assembling drones at NXP ourselves, which took a lot of time. Then Jari van Ewijk, electrical engineer from Blue Jay turned up and built one in under 90 minutes. I asked how he could do it so quickly. He said ‘how many prototypes you think I’ve had to build? We’ve crashed quite of few of them…’ He could probably have done it blindfold! The next day we received pictures and movie clips of their first flight with that NXP-drone kit!”
Another spin-off from Blue Jay is that the technology involved can have applications beyond the narrow (albeit very important) fields of life-saving and hospitals. “For example, there’s huge potential for FMU technology and drones in areas like warehouse logistics,” says Gino. “So the beta-testing also contributes to making this a proven solution NXP can offer in other contexts and sectors where the same parameters apply.”
Priceless know-how
Fourtress provide software solutions for technical systems. “We’ve been involved since the first Blue Jay team,” says field manager, Douwe Gerritsen. “Initially, we simply provided technical knowledge. But as we have engineers with pretty strong drone-related experience, for the past two years instead of money we have loaned the team an engineer for 3 months.”
It’s win-win as the engineer also gets interesting experience. He has worked on the low-level systems, Lidar, obstacle detection, the drone’s inner workings, the ultra-wide band and data communication systems. Fourtress also provided a student on placement with them to work on the Blue Jay mobile app development and drone-API as their thesis project.
“Probably our biggest challenge in developing the app is that Blue Jay team projects are very dynamic,” says Douwe. “They set ambitious goals. That’s inspiring, of course, but means implementation tends to change along the way as they encounter problems. You try to work with that flow, so you don’t become an obstacle yourself. But sometimes you need to help them set boundaries in terms of what’s achievable within the timescale.
“There’s also the technical challenge of trying to fly a drone in populated environments, including the paradox of wanting a drone that both works totally autonomously, but is also controlled by people.” This means a person must be able to use the app to give the drone commands, after which the drone figures out for itself how to execute the task. Sometimes deciding autonomously, for example, which route to take or task to prioritize. But the interface needs to let people easily set new tasks and get a clear overview that updates them on what each drone is doing at that moment.
“The app is basically the drone’s manager. It gives the drone tasks, but the drone decides how to execute them. The prioritization aspect is interesting. It’s important the user can schedule tasks to be done later or immediately. But also, that they get feedback on how quickly the drone will actually do the task. Because when an emergency happens it will override everything else. This variety of tasks makes things a lot more complex.”
Another challenge is notifications. “The tricky part is that people will use their phones not just to control the drone, but also to surf the Internet, take calls, etc. Technically, this means the app must be able to run a service in the background even when it’s not active, so it can still give users notifications, for example of an emergency or a completed task, when their drone app isn’t active.
Douwe sees the Blue Jay partnership as an invaluable one for Fourtress. “How we operate with Blue Jay is a bit of a model for how we try to operate with all our customers. In that, if you hire one Fourtress person, you hire all of Fourtress: all our knowledge is at your disposal.”
Professional fans
Blue Jay gets students out of their ivory tower and working with real-world organisations to address real-world needs. So it’s reassuring when society responds so positively. “The Reddingsbrigade were very enthusiastic from the start,” recalls Mike. “It’s cool how they’re so open to adopting new technology. Ultimately, we share the same goals. By working together, we can help each other achieve those goals better and quicker.”
At IBM, Arjen has quickly become a huge fan of the students and what they’re doing. “Like much of my own work, this project is at the forefront of technology. I bring my experience to the project, but these young people have a different perspective and come with fresh ideas, so we definitely learn from each other.” Douwe echoes this view on behalf of Fourtress. “Because with Blue Jay you’re working with such enthusiastic and dedicated people, it makes it so much fun.”
At NXP, Gino is particularly impressed by how the Blue Jay team leverage maximum value from partners’ expertise and function so maturely as a unit. “They take technology from us for the drone, from IBM for in the cloud, use other partners for the mechanical chassis… operating just like an efficient little company, each with their own areas of responsibility. Of 18 team members, I only deal with 2 or 3 hardware engineering, electronics and software people. For young, inexperienced students it’s pretty impressive.”
With her hands-on experience, LOTUS’ Anja is equally excited by the drone’s potential. “It’s basically an extra pair of eyes. Even the best lifeguard gets tired, but not the drone. It stays alert, and that means once it’s fully operational it will help ensure people are safer.”
Those days are still a way off. For now, it’s all about proof of concept. Even so, the team will continue to work on a drone that can provide first aid in emergency situations. Because, as Arjen says, “if one day the drone saves just one life, the whole project will have been worth it.” And you don’t need AI to detect the truth in that statement.
Learn more about Blue Jay Eindhoven: Blue Jay Eindhoven
Or get in touch with the team if you’re interested in supporting their work, getting involved or becoming a partner. Send an email to: info@bluejayeindhoven.nl
Martijn is based in NXP’s Eindhoven headquarters where he’s been an integral member of NXP’s corporate communications team since 2007. He spearheads internal and external PR and communications initiatives, as well as NXP corporate sponsorships with university partners and policymakers.
