Facial Recognition and Assistive Technology

You’ve explored what assistive technology is. Now, let’s give your Micro:bit a “brain.” You will teach a computer to recognize your unique actions—like a smile, a head nod, or a specific sound—and use that to send a wireless signal to your Micro:bit, turning it into a truly smart device.

Today’s Goals

  • Program your Micro:bit in MakeCode to react to a wireless signal.
  • Train a custom AI model to recognize your unique gesture, expression, or sound.
  • Connect your AI model to your Micro:bit, making your action trigger the device from across the room!
  • Brainstorm how this AI-powered “switch” can be used in your own assistive technology prototype.

Materials You’ll Need

  • A computer with a webcam and microphone 💻
  • Your Micro:bit with battery pack and USB cable
  • Prototyping materials (cardboard, foil, tape, etc.)

Part 1: Train Your AI “Switch”

Now, let’s create the AI that will send the signal.

  1. Open Teachable Machine: Go to Teachable Machine and start a new Image Project (for gestures/expressions) or Audio Project (for sounds).
  2. Create Two Classes:
    • Class 1: Your “ON” Switch. Name this after your chosen action (e.g., “Smile,” “Head Tilt Left,” or “Ahh Sound”). Record at least 100 examples.
    • Class 2: Your “OFF” Switch. Name this “Neutral.” Record yourself looking straight ahead or just record the quiet background of the room.
  3. Train and Export: Click “Train Model.” When it’s finished, click “Export Model” and then “Upload my model.” Keep the sharable link that it generates open in a new tab. You will need it in the next st
Part 2: Program Your Micro:bit’s Reaction

First, let’s program your Micro:bit to listen for specific commands using Bluetooth. This will allow your computer to send signals directly to your device as if they were connected by an invisible wire.

  1. Open MakeCode & Add Bluetooth Extension:

    • Go to the Microsoft MakeCode for Micro:bit website and start a new project.
    • Click on “Extensions” at the bottom of the block menu.
    • Select the “Bluetooth” extension. You will see a warning that Radio blocks will be removed. Click “Add extension” to continue. You will now have a blue Bluetooth category in your menu.
 

  1. Show Connection Status:

    • From the Bluetooth drawer, drag out an on bluetooth connected block.
    • From the Basic drawer, place a show icon block inside it and choose an icon that means “connected,” like a diamond.
    • Do the same for the on bluetooth disconnected block, but choose an icon that means “disconnected,” like a small square.

  2. Start the Right Bluetooth Service:

    • Drag the on start block to an empty part of your workspace.
    • From the Bluetooth drawer, find the bluetooth uart service block and drag it into the on start block. This special service allows the Micro:bit to send and receive text strings over Bluetooth.

  3. Create a Variable to Store the Signal:

    • From the Variables drawer, click “Make a Variable…”.
    • Name the new variable receivedString and click OK.

  4. Build the Main Command Listener:

    • From the Bluetooth drawer, drag out the main event block: on bluetooth data received new line (). This block will run whenever the Micro:bit receives a text command that ends with a “new line.”
    • Inside this block, go to Variables and drag in the set receivedString to block.
    • From the Bluetooth drawer, find the bluetooth uart read until new line () block and snap it into the set block. Your code now reads the incoming text and saves it to the receivedString variable.

  5. Program the Actions:

    • From the Logic drawer, drag two separate if...then blocks and place them one under the other inside the on bluetooth data received block.
    • First Condition: For the first if, go to Logic and drag a 0 = 0 comparison block into the condition slot. Change it to look like this: receivedString = "Yes". (You’ll find the receivedString block in Variables).
    • Inside this if block, place a show icon block and choose a “check mark” or “happy face” icon.
    • Second Condition: For the second if, build the condition receivedString = "No".
    • Inside this if block, place a show icon and choose an “X” or “sad face” icon.
    • Third Condition: For the second if, build the condition receivedString = "Nothing".
    • Inside this if block, place a show icon and choose an “Umbrella” or “sad face” icon.
    •  

    Your final code should look identical to the one in the image below!

6. Download the Code:

    • Click the “Download” button. MakeCode will ask you to pair with your Micro:bit.
    • Connect your Micro:bit to the computer with your USB cable and follow the on-screen instructions to pair it.
    • Once paired, click Download again to transfer the program to your Micro:bit.

Your Micro:bit is now a smart device, ready and waiting for the “Yes” or “No” or “Nothing” commands from the AI you’ll build in the next activity.

Trouble with your code? Try mine! https://makecode.microbit.org/S97652-06532-55963-10690

Part 3: Connect AI to Your Micro:bit

This is the bridge between your AI model in the cloud and the physical Micro:bit in your hand. We will use your webcam that send the correct commands to your device over Bluetooth using the cardboard.lofirobot website.

  1. Go to the cardboard.lofirobot Website:

  2. Load Your AI Model:

    • Find the shareable link you got from Teachable Machine in Part 1 (see image below for link).
    • Paste this link into the text box labeled “Teachable Machine URL”. Your class names (e.g., “Smile,” “Neutral”) will appear below.

  3. Connect Your Micro:bit:

    • Make sure your Micro:bit is plugged in and showing the “CheckerBoard” icon from the code you downloaded.
    • On the website, click the blue “Connect Micro:bit” button.
    • A Chrome pop-up window will appear and start scanning. Select your Micro:bit from the list and click “Pair”. If successful, the button on the website will turn green and say “Micro:bit Connected.”
    •  

  4. Test Your Creation!

You have successfully created a wireless, AI-powered switch for your assistive technology prototy

Made with Padlet
Previous Lesson
Back to Workshop
Next Lesson