- A Tour Inside Arduino Core: Source Files, How to Make A New Core and Arduino Building Steps
- A Tour inside Arduino Core
- Generic Source Files
- Source Files Related to Digital and Analog Ports
- Source Files Related to Timing and Timers
- Source Files Related to Hardware Serial (mostly)
- Miscellaneous Source Files
- Files System Tour
- How to Make a New Arduino Core
- Adding New Arduino Core to IDE Offline
- Arduino Building Process
- Read More
A Tour Inside Arduino Core: Source Files, How to Make A New Core and Arduino Building Steps
Diving into an open-source project code/design brings a lot of benefits to the diver especially if it’s a global and collaborative project which means that the diver will be exposed to different mindsets and smart hacks. Definitely, Arduino project is one of these projects. What adds even more uniqueness to it, is the new way of modularity and abstraction proposed in Arduino for Hardware (MCU world particularly) which was uncommon in the past. Arduino wants the developer’s energy to be focused on solving the problem rather than reinventing the wheel by doing some low-level programming and writing a lot of boilerplate code to control the MCU’s hardware. A Lot of developers hate Arduino style, and hate how a novice who doesn’t even know how registers work can program an MCU. Arduino made it very easy to make a full project without dealing with one single register. For developers intended to be experts or students from engineering background, it’s not good to keep their eyes blind about low-level details because neither low-level bugs nor the application can be done effectively without knowing and mastering the details. Personally, my old journey with Arduino began without knowing any low-level details but I had a personal eager to know the background magic behind the scenes. While my electronics experience was growing I could understand the Hardware well , and dealing with MCUs with bare-metal C in addition to dealing with more than one MCU family made understanding the software (Arduino core for MCU) possible. Arduino core is the place where all built-in functions like: pinMode(), digitalWrite(), analogRead(), Serial.begin() …etc are defined. This article is an attempt to understand the Arduino core structure, and even building a clone. Moreover, it also discusses the building process of Arduino sketches.
[UPDATE] Arduino has now its own official documentation of platforms. Also, one from the Arduino community reviewed this article and has added important notes.A Tour inside Arduino Core
As we said, Arduino core is the place where all built-in functions source files can be found. These tables summarize the most glaring things inside most of core source files grouped according to the functionality.
Generic Source Files
| Source File | What Is Inside? |
|---|---|
| Arduino.h | 1- Declaration of most built-in functions of Arduino like pinMode(), digitalWrite(), . etc. 2- Macros of some constants like HIGH, LOW, INPUT, OUTPUT, ..etc. 3- Macro functions for bitwise operations and some other general operations like: min(), rand(), ..etc. 4- Declaration of pin and port mapping arrays(check our micro-blog about mapping arrays in Arduino) in flash memory to make a map between Arduino pin number and the physical number. I.e. mapping pin 13 to the according port and pin in the MCU registers. The value of these arrays can be found here: hardware\arduino\avr\variants\standard\pins_arduino.h |
| main.cpp | Here where the basic structure of Arduino program is declared. Here is the actual program: int main(void) init(); initVariant(); setup(); for (;;) loop(); > return 0; > |
Source Files Related to Digital and Analog Ports
| Source File | What Is Inside? |
|---|---|
| wiring_digital.c | Defines digitalWrite(), digitalRead(), pinMode(), ..etc. |
| wiring_analog.c | Defines analogRead(), analogWrite() ..etc. |
| wiring_shift.c | Two functions to read in/out serial bits-streams. |
| wiring_pulse.c | Two functions to read input PWM signals |
| WInterrupts.c | Defines some functions to handle external interrupts. |
Source Files Related to Timing and Timers
| Source File | What Is Inside? |
|---|---|
| wiring.c | 1- Defines timing functions like millis(),delay() ..etc. 2- Timer0 overflow ISR 3- Initialization function used to initialize some peripherals before calling setup function in main.cpp. |
Source Files Related to Hardware Serial (mostly)
| Source File | What Is Inside? |
|---|---|
| Stream.h, Stream.cpp | Defines the Stream class which its methods are used (by inheritance) in HardwareSerial class like readUntil(), find(), ..etc. |
| Print.cpp, Print.h | Defines the Print class with methods to print messages (print & println). This is inherited by the Stream class. |
| Printable.h | This header defines a class that can be inherited by others to make any new object type printable by print() and println(). Check our micro-blog about Printable class. |
| HardwareSerial.h, HardwareSerial.cpp, HardwareSerial_private.h | 1- Defines the HardwareSerial class. 2- Other methods like print(), find(), ..etc are inherited from Stream class. |
| HardwareSerialx.cpp | 1- Defines UARTx ISR. 2- Declares the class object HardwareSerial Serialx. |
Miscellaneous Source Files
| Source File | What Is Inside? |
|---|---|
| New.cpp and new.h | New definitions for dynamic memory allocation functions. |
| binary.h | Macros of Bxxx notation in Arduino |
| Tone.cpp | Some functions to generate tones on some pins. |
| WCharacter.h | Defines some simple functions to process characters like: toAscii() using some function from ctype header. |
| Wmath.cpp | Some math functions like map() function. |
| wiring_private.h | Some scattered declarations. |
| WString.cpp & WString.h | Defines the String class in Arduino. |
Files System Tour
The tour will focus on the part related to Arduino core only and the files related to the building process. First, all source file of Arduino core is placed inside the following directory \Arduino\hardware\arduino\\cores\arduino. Another place for Arduino third party cores (i.e. ESP8266 core) is in \Users\\AppData\Local\Arduino15\packages and these cores are the ones downloaded from board manager (Tools>Boards>Board Manager). Tool-chain tools (and even avr-gcc header files) are located in Arduino\hardware\tools\ and here Users\\AppData\Local\Arduino15\packages\arduino\tools.
How to Make a New Arduino Core
As an example, a clone for avr core will be made. To make a new Arduino core you need two main things: 1- JSON file to describe the new core: its name, versions, boards, tools dependencies, URLs to download files … etc. 2- Core source files packed in .zip, .tar.bz2 or .tar.gz file.
First, The JSON file name should be like “package_x_index.json” (or any other descriptive name) and contains the following template:
< "packages": [ < "name": "aaa", "maintainer": "bbb", "websiteURL": "http://bbb.com/", "email": "aaa@bbb.com", "help": < "online": "http://bbb.com/" >, "platforms": [ < "name": "aaa Boards", "architecture": "avr", "version": "1.0.0", "category": "aaa", "url": "http://bbb.com/aaa.zip", "archiveFileName": "aaa.zip", "checksum": "SHA-256:x", "size": "x", "help": < "online": "http://bbb.com/" >, "boards": [ < "name": "Arduino/Genuino Uno" >], "toolsDependencies": [ < "packager": "aaa", "name": "avr-gcc", "version": "4.9.2-atmel3.5.3-arduino2" >, < "packager": "aaa", "name": "avrdude", "version": "6.3.0-arduino6" >] > ], "tools": [] > ] >
Now, change the JSON file information according to new core details. There are many online tools to calculate the SHA-256 checksum of core zip file like this one. Finally, Try to check other cores json files like esp8266 core JSON file. The next step is to make the zip file. It should include one folder with the core name (aaa) and the core “template” file system inside (bootloaders, cores, doc, libraries, tools and variants folders and boards.txt, platform.txt and programmers.txt files). As this article is about an overview not an in depth about making new core, you can check the job of each one of these files from the formal documentation.
platform.txt contains definitions for the CPU architecture used (compiler, build process parameters, tools used for upload, etc.) boards.txt contains definitions for the boards (board name, parameters for building and uploading sketches, etc.) programmers.txt contains definitions for external programmers (typically used to burn bootloaders or sketches on a blank CPU/board)
name=Atadiat AVR Boards version=1.0
# See: http://code.google.com/p/arduino/wiki/Platforms menu.cpu=Processor ############################################################## uno.name=Arduino/Genuino Uno uno.vid.0=0x2341 uno.pid.0=0x0043 uno.vid.1=0x2341 uno.pid.1=0x0001 uno.vid.2=0x2A03 uno.pid.2=0x0043 uno.vid.3=0x2341 uno.pid.3=0x0243 uno.upload.tool=avrdude uno.upload.protocol=arduino uno.upload.maximum_size=32256 uno.upload.maximum_data_size=2048 uno.upload.speed=115200 uno.bootloader.tool=avrdude uno.bootloader.low_fuses=0xFF uno.bootloader.high_fuses=0xDE uno.bootloader.extended_fuses=0xFD uno.bootloader.unlock_bits=0x3F uno.bootloader.lock_bits=0x0F uno.bootloader.file=optiboot/optiboot_atmega328.hex uno.build.mcu=atmega328p uno.build.f_cpu=16000000L uno.build.board=AVR_UNO uno.build.core=arduino uno.build.variant=standard ##############################################################
3- Making the zip file “atadiat.zip” and calculating the checksum using SHA online generator. 4- Making the JSON file according to the mentioned template. 5- To test this new core, Add JSON URL to Arduino IDE (https://atadiat.com/wp-content/uploads/package_atadiat_index.json) 
Then go to board manager and download the new core. 
Adding New Arduino Core to IDE Offline
If there is a need to add a core without hosting the core and JSON file on a website. The easy way to do that is to create a file with the core name in the following directory Documents\Arduino\hardware. The full directory would be like: Documents\Arduino\hardware \\
Arduino Building Process
Arduino native IDE has cryptic compiling-and-linking process which means that you can’t find a makefile defines the whole process of building code. Instead, the platform.txt file can give the developer the ability to change the building process a little, but some stages of building process will still be out of the developer control. Keep in mind that some information are retrieved from board.txt and used while the building process. Building process in Arduino (for AVR architecture for instance) consists of the following stages in order ( to double check that, enable the output messages during compilation):
- The IDE detects the used board and core according to user choice and retrieves the corresponding information form board.txt.
- A temp directory is created in the following path Users\\AppData\Local\Temp\arduino_build_x to store all compiled and temporary files in it. Object files for all Arduino core source code files and used libraries files will be there beside the output files of compilation like HEX file which will be uploaded later to the MCU by the programmer.
- Compiling sketch.
- Compiling the libraries source files.
- Compiling Arduino core source files with the following steps:
- Compiling core C files.
- Compiling core C++ files.
- Compiling core assembly files.
In a nutshell, what Arduino allows you to do with platform.txt is:
- Defining your own compilation/building flags/variables to add them later to building commands.
- Changing the “pattern” of each stage mentioned previously.
- Changing building commands.
- Adding hooks to the building process.
Have a look how Arduino building patterns look like:
## Compile c files recipe.c.o.pattern="" -mmcu= -DF_CPU= -DARDUINO= -DARDUINO_ -DARDUINO_ARCH_ "" -o "" ## Compile c++ files recipe.cpp.o.pattern="" -mmcu= -DF_CPU= -DARDUINO= -DARDUINO_ -DARDUINO_ARCH_ "" -o "" ## Compile S files recipe.S.o.pattern="" -mmcu= -DF_CPU= -DARDUINO= -DARDUINO_ -DARDUINO_ARCH_ "" -o ""
These patterns are converted to repeated building commands
Thankfully, Arduino team has added since IDE 1.6.5 a way to add building steps as a hook to the predefined building process called pre and post build hooks.
recipe.hooks.sketch.prebuild.NUMBER.pattern (called before sketch compilation) recipe.hooks.sketch.postbuild.NUMBER.pattern (called after sketch compilation) recipe.hooks.libraries.prebuild.NUMBER.pattern (called before libraries compilation) recipe.hooks.libraries.postbuild.NUMBER.pattern (called after libraries compilation) recipe.hooks.core.prebuild.NUMBER.pattern (called before core compilation) recipe.hooks.core.postbuild.NUMBER.pattern (called after core compilation) recipe.hooks.linking.prelink.NUMBER.pattern (called before linking) recipe.hooks.linking.postlink.NUMBER.pattern (called after linking) recipe.hooks.objcopy.preobjcopy.NUMBER.pattern (called before objcopy recipes execution) recipe.hooks.objcopy.postobjcopy.NUMBER.pattern (called after objcopy recipes execution) recipe.hooks.savehex.presavehex.NUMBER.pattern (called before savehex recipe execution) recipe.hooks.savehex.postsavehex.NUMBER.pattern (called after savehex recipe execution)
As an example we will add a hook after core compilation. First, we will add some variables (variables name can be anything else):
compiler.hook.cmd = avr-gcc-nm compiler.hook.cmd.flag = -g
The hook here is to list symbols inside an object file using a tool from GCC called nm. Second, we will add the hook:
recipe.hooks.core.postbuild.1.pattern = "" "/core/wiring_digital.c.o"
Note: Multiple hooks can be added to the same stage by increasing the hook number.
Finally, If the developer needs even more control to the building process, then projects like Arduino-makefile should be considered. Arduino-makefile defines entire workflows for compiling code that can be customized.
Read More
