Logger Project

Further adventures in Td5 Map Programming

Sunday, June 18, 2023 - 17:15

There has been a bit of progress since the last post.

The module videos don't really give a good sense of how the app works, so this time I've tried to give some idea of how things are bolted together.

The "Map Tools" section of the app deals with ECU firmware. The flow through the modules in this section is input -> information -> output.

Input options are: - Map Finder: new firmware from database - Open File: read firmware from file (Nanocom .map, 16kb bin, 256kb bin, 256kb Kess bin, MEMS3 Flasher Map and Firmware formats) - Read from ECU

Output options are: - Write file: write to any of the above formats. - Write ECU: map programming.

This allows the app to work as a "map wizard", file format convertor and programmer.

The video shows the Map Finder's option to display maps that suit the connected ECU.

The programming module defaults to programming the tune only if the variant code already exists on the ECU. In these cases "Overwrite Variant" gives the option to program the variant

As a bonus there is a quick tour of the Injector programming module. There isn't much to see - it edits and programs codes.

The app is currently running on macOS and Windows 10+.

I'm having some issues with PyInstaller built .exe files being identified as malware. These are false positives and its a known issue which seems to require an expensive code-signing signature to solve.

ECU Read and Save

Friday, September 3, 2021 - 19:00

This is a short demo of the ECU read and file save functionality of an app I'm currently working on.

The app is designed with Td5 maps in mind.
By default downloads target the exact length of the stock tune and optionally variant.
This behaviour can be overridden by using the Read Full Blocks option which forces reads of entire flash content for both tune and variant maps.
In most cases a minimal read of the tune is all that is required.

The progress bars adjust depending on selected options.
Progress reflects the percentage read of the total download.

One of the features is the ability read a 16kb tune and save a full 262kb bin with correct boot map and variant. Programming record is update with the current date and metadata for the loaded tune.

Datalogger Project - XDL logging milestone.

Saturday, August 3, 2019 - 16:45

The past week has seen a lot of progress on the Datalogger.

The event-driven non-blocking FAT32 library I posted about earlier in the year was integrated into the logging module and after some initial hiccups is now working well.

CSV logging to card is now running well. Auto detect of MSB // NNN hardware has been implemented and throttle mapping switched on that basis. MSB ECU's report two throttle tracks, throttle %, and supply voltage, while NNN ECU's report three throttle tracks.
A diagnostic request that returns the raw ADC conversion values, and fault codes are also dependent on ECU hardware type.

EGT probe support has been integrated into the CSV logging. The logger checks if to see if there is a probe connected at the start of each log run, and adds the EGT column if required.

The big news - for me anyway - is that logging to Tuner Pro XDL format is working. The code for the XDL log format was written yesterday - based on the reverse engineering done for the Nanocom conversion as posted previously - , and a small bug in a log buffering code which prevented the logs writing to card was fixed this morning.

I've only had a chance to do a few short runs around home - lots of 40kph zones and slow moving traffic. Aside from my laptop going to sleep after 15 minutes and killing the longest of the logs, the logging seems to be pretty stable.

This evening I looked up my test ECU and eft the logger run on the bench recording to XDL for a bit over three hours without any issues.

As a bit of a teaser, this is a screenshot of Tuner Pro RT tracing from a log I recorded today.
The black spots on the tables indicate the current running point.

XDL Datatracing

None of the parameters being used are calculated - it's all logged ECU data.

The to-do list is still pretty long, but I'm now feeling far more optimistic than I was 3 months ago.

Data tracing in Tuner Pro...

Tuesday, April 9, 2019 - 08:00

As I posted recently the data logger project has been taking up a lot of my time over the past few months.

The main intention of the logger is to enable faster, configurable data logging which can be saved into either CSV or Tuner Pro XDL formats.

Over the past few weeks I've been doing a bit of work on reverse engineering Tuner Pro's XDL data log format. I got about 95% worked out before I got in touch with Tuner Pro developer Mark Mansur who very kindly pointed out with the parts I'd got wrong and filled in the bits I couldn't figure out.

As a way of developing and testing the ADX writer code on the desktop, I've written a bunch of code that reads in an ADX file and Nanocom log file (from an Nanocom running current firmware) and uses that information to create an XDL log file.

There are some issues like the lack of Throttle % in the EVO logs that makes the convertor less useful than it should be.

As a teaser to show the possibilities ADX logging opens up this screen capture shows TP data tracing a converted Nanocom EVO log.
The MAF airflow calculation is being done in Tuner Pro RT, and a simple monitor setup shows AAP and MAP.

Datalogger - the rebooted reboot...

Tuesday, March 12, 2019 - 15:45

While things have been fairly quiet on the site recently, that hasn't really been a reflection of the level of work that has been going on at the DiscoTd5 Lab for last three or four months...

The datalogger had pretty much been dumped in parts bin as the level of work required to get it up to a useful state seemed to be overwhelming. Conversation with an enthusiastic D2 tuner convinced me to take another look - on the basis of build it or bin it for good.

Three months later, and hundreds of hours rewriting the code, it's slowly starting to come together.

The biggest chunk of effort - which has basically eaten the last 4 weeks - has gone into writing a SDCard // FAT32 library from the ground up. This was the obstacle that had resulted in the project being shelved previously so it's been a significant hurdle to overcome. There is still work to do but there is now light at the end of the tunnel.

It might seem a bit bizarre to be writing a FAT32 library from scratch when there are well established products like the ELM Chan FATFs available.
The main issue has been that the logger code is written in a way that requires components to be non-blocking. Blocking code will prevent any other code running while the blocking waits in a loop for something to happen - it literally blocks progress.
ELM Chan's FATFs is blocking in that it waits for SDCards to come out of busy state before proceeding. The issue here is that at times cards can be busy for extended periods - 50ms is not unusual even with a good quality card like a SanDisk Ultra. With poor quality cards wait times of 200ms+ are not uncommon.

Where this becomes a problem for a Td5 data logger is that the default time between an ECU response and the next "tester" request is 25ms.
The Nanocom Evo solves the problem by making a series of requests - one log line of data - then writing that line to file. There is a 600ms+ pause between the end of one block of data, and the start of the next which is plenty of time for even the slowest junk SDCard to finish writing and return to idle state. But this also means that the Nanocom is writing one line of log data every 1.25 seconds. Fine for some purposes but a lot of dynamic behavior "falls through the cracks" - the torque reduction of an auto shift completes in under 300ms for example, so is effectively invisible.

The approach I'm taking here is to use non-blocking code with DMA (direct memory access) transfers of data to the card. The DMA transfers effectively run in the background without processor intervention, so the combination means that the processor only needs to setup the write to the SDCard, and then can resume more important tasks like reading data from the ECU, or other sensors. The issue of wait times is dealt with by buffering the data waiting to be written to card. In theory all this will allow data rates 10-15 times faster than the Nanocom to be written to card without delays caused by slow card writes.

An excuse for new toys

One of the issues I'd have was that my old Saleae Logic - an original and now much cloned 8 channel 24Mhz job - is starting to show it's limitations when trying to capture communication with the SDCard at 20Mhz bus speed. To capture a signal you need at least twice the frequency of the signal - so 40Mhz at least - ideally you oversample at x4 so 80Mhz. With the old Logic I was flying blind.
As much as I love the Saleae and the software, a new Logic 8 Pro which has sufficient speed for this task was prohibitively expensive - 700-800USD.

So I've ended up with an open source design analyser that does either buffered or streaming captures. The buffered captures are made to internal memory, and have fairly limited length of capture. around 1 second @100MHz using 4 channels, and 300ms @ 400Mhz using 4 channels. Streaming captures are sent over usb to an open source, cross platform application based on Sigrok. The buffered captures are limited to 100Mhz with 3 channels, or 50MHz when using 6 channels. For the SDCard work I need 4 channels and long captures so need to use the 50MHz/6 Channel option. So not perfect but it's flexible enough to be quite useful. Best thing was that it's far more affordable at $150USD.

DSLogic Plus vs Saleae Logic original
The Logic is the small black square on the left, the DSLogic Plus on the right is significantly bigger but still fairly compact.

And the DSLogic proved it's worth within an 30 minutes of unboxing. I'd had an issue where the code would run perfectly with the Saleae connected but would then fail as soon as I disconnected.
It's a real catch22 - the bug occurs only when your debugging gear isn't connected... makes troubleshooting fun. The DSLogic has coaxial probe leads which significantly reduce the effect on the signal being monitored compared with the Saleae's flying leads. There is a great comparison of a Saleae clone and DSLogic on the OpenTechLab channel on YouTube - the sims of the effect of flying leads is quite an eye opener.

The video above mentions the probe clip quality. The ones that came with mine seem to be cheap knock-offs of ez-hooks. Not quite as clunky as those seen in the video, but hook wire is pretty flimsy compared with ez-hooks.

The upshot was that even with the DSLogic connected the "undebugable" bug was occurring every time a card was inserted, and it was straight forward to isolate where and why the code was failing.

DSView

The software is fairly good but a bit rough around the edges at least under macOS. I have a bit of trouble with zooming using the trackpad - it's clearly designed for left and right buttons plus scroll wheel. Protocol decoding seems accurate but is slow compared with the Saleae app, although to be fair10 second captures at 20Mhz were regularly bringing that program to the point where a force quit was required, and in one instance a full reboot.

For the money it is pretty decent, and the cross platform software is a big plus in my book.
Worth noting that the DSLogics sold on eBay are not covered by a manufacturers warranty.
I opted to purchase direct.

Datalogger - more updates

Tuesday, July 25, 2017 - 15:00

It's been about six weeks since I posted the last update to the site, so thought I better post something...

Since the last post I've been working on implementing a modular system, with the idea that the basic logger can handle control inputs from USB, Bluetooth LE, a simple stop/go button interface, or a touch panel. The "Pass Through" interface shown in the last post is intended as a way of allowing applications running on mobile devices to communicate with the ECU without the need to micro-manage the low level protocol. You send a Service ID, Resource ID and any other necessary data for the request and get back a positive response plus data, or negative response if the request fails.

The second module is a configurable logger. And this is what I've been working on since start of July. The logger is currently set up with definitions for the 18 or so diagnostic requests that I think you'd realistically want to log.
Requests are set by using something like..

AT M LC  09 0D 1B

LC indicates a "Log Configure" command, and the items to log are specified using the Request ID. In this example 0x09 = RPM, 0x0D = Road Speed, and 0x1B = Throttle Voltages. The command input is not case or space sensitive.

Starting and stopping logging is controlled by AT commands.

AT M LS  
AT M LE  

These are the "Log Start" and "Log End" commands respectively.

By default the data from the logger is in ECU format with no transformations. Using the LR command

AT M LR  

toggles between "Raw" and "Convert" mode which converts Kelvin to Celsius and shifts decimal places where appropriate.

The next thing I need to tackle is the logging to SD Card and the button control interface.

Datalogger updates

Sunday, April 24, 2016 - 14:45

After a week or two distracted by the OLED graphic display - which turned out to be damaged in the end - I've refocused on the core logger component.

I'm now using a simple LED which changes state every time a new line is written to the log buffer. Currently this occurs at slightly less than 0.5 second intervals so it's very obvious when logging stops.

The "refocusing" has meant I've found and squashed a couple of bugs that were causing intermittent issues with the log files. I've also discovered that the road speed data from the ECU is intermittently transmitted with an incorrect checksum. Until I found this issue I had assumed that I was logging too fast. Now that I've added handling for incorrect checksums the diagnostic connection will keep running after "ignition off" until the ECU shuts down.

With a bunch of bugs fixed the logger had a short road test this afternoon. In previous tests the connection would drop after 3 minutes maximum. This time around I was able to take the D2 for a 15 minute drive including a short run on a nearby freeway without out a hitch. I did have a moment of fuel cut as I accelerated on to the freeway and the logs show clearly the MAF sensor voltage climbing above the 4950mV upper limit, and the MAF reading dropping to 0.

A second run to work, to pick up a few things gave another opportunity to test the stability of the logger. The round trip was just under 30 minutes and I found I was able turn the motor off, switch back to accessory position, grab the things I needed from the office, then restart the engine and drive home without dropping the connection. The Nanocom is lucky to do the trip one way without throwing in the towel.

This is a quick comparison of the "Airmass per cycle" figures the ECU calculates from the MAF and MAP done using the MegaLog Viewer scatter plot function.

Td5 Datalogger Reboot

Monday, March 28, 2016 - 18:00

After a long hiatus I've been working on getting the Td5 Data Logger (aka Raijin Project) up and running.

The low level connection management code has been rewritten so it's a lot more robust. One of the big improvements is that it now uses the 5ms minimum inter byte timing specified by ISO14230 - in practice it's actually between 4.25 and 4.75ms. My theory is this timing allows the ECU more time to handle core processes so should help prevent dropped communications sessions under load. It's still missing error handling and recovery but otherwise working well.

The point to point wired board I've been using for development was a bit impractical for in-car testing, so I've had a piggy back board made up which should make it slightly easier to work on. The board is designed to plug on to a Fubarino SD development board, and has a ISO9141 interface chip, k-type thermocouple interface, 5V voltage regulator running off the OBDII port supply, a 5 way tact switch, and a connector for a small 0.96" OLED. The whole thing is about 25mm by 75mm.

I got notification today that the test boards have finally been shipped, a week later than the original estimated shipping date. I'll be interested to see how they turn out as this was the first time I've used Eagle to design a board and the first time I've had one of my own designs fabricated.

Td5 Datalogger

Tuesday, July 15, 2014 - 17:30

While the Nanocom is a pretty useful diagnostic tool it leaves a lot to be desired in terms of datalogging, so I've been messing around with the idea of building a dedicated datalogger for the Td5.

The time I've put into reserve engieering the ECU code has been very useful in this regard. I have been able to identify the routines used to generate and verify the security keys - done using linear feedback shift registers - which are required to gain access to many of the diagnostic functions of the Td5 ECU. 

At this point I've been able to sucessfully negotiate a connection with the ECU and request "live" data. On my test ECU running on the bench I'm able to request a single parameter every 20ms. Unfortunately this seems to take a bit of a hit when the ECU is actually hooked up to a running engine, so this appears to best possible under ideal conditions.  Nonetheless it appears that it should be possible to log full engine data at around 3-5 times per second, which is substainally better than the Nanocom's once every 1.1 seconds. By prioritising requests so that slow changing data is requested less frequently it should be possible to log critical parameters 10 times per second or faster.

I've also verified that it is possible to request specific data from the ECU providing you where in memory the data is stored. This opens up the possibility of logging the values returned for ECU map lookups.  As an example you could log throttle amount, aircharge and the result of the smoke map lookup.

It's still early days yet and I'm currently working to make the code I have already written more robust.

The current wish list also includes an  interface for a EGT probe, and ability to connect of other sensors that output 0-5V.

I've attached a screen capture showing a capture of "Raijin" ( the shinto god of storms and thunder) sending a key response and the ECU responding with a security access OK message.  Top line is the transmit line and the bottom is the recive line. The ISO interface chip (ST-Micro L9637) I'm using echoes the transmitted data back on the rx line hence the duplication of the TX packets.

 

Security access granted response from ECU

 

randomness