Kitesurfing - Downhill - 21 Sept 2018

Building MicroPython for esp8266 - to freeze modules

For my esp8266 project with the ssd1306 screen I was starting to run out of memory for all the 'threads' I needed. I have a TCP client talking to the server, temperature readings coming from the 1-wire device, and button inputs. Not to mention that I want to run a webserver for configuration.

The answer, and it has so far worked very well, is to build my own firmware, and 'freeze' the modules that are least likely to change into the firmware. I have a font class for the ssd1306 screen, the asyncio library, the aswitch library, and a few others which are unlikely to change, so I picked them.

This is the procedure I followed.

Create a VirtualBox VM, running some recent version of Ubuntu (18.04) (select minimal options), and log into it. I installed opensshd-server, and on the settings for the VM I enabled a bridged network so it gets an IP address on my home router.
sudo apt-get install net-tools openssh-server
Then you can ssh

I more-or-less follow this : http://akshaim.in/IoT/MicroPython_ESP8266/MP_ESP_101.html


sudo apt-get install make unrar-free autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python-dev python python-serial sed git unzip bash help2man wget bzip2 libtool-bin

git clone --recursive https://github.com/pfalcon/esp-open-sdk.git
cd esp-open-sdk
make

(takes about 40 minutes)
(NOTE - this path is shown at the end of the install - run the command)
export PATH=/home/akshaim/Documents/ESP8266/esp-open-sdk/xtensa-lx106-elf/bin:$PATH




git clone --recursive https://github.com/micropython/micropython.git
cd micropython
git submodule update --init
make -C mpy-cross
 cd ports/esp8266
(different from original instructions)
make
(different from original instructions)

Now you can put modules (and subdirectories) into the modules directory, and they end up in the firmware. When you put a new module, you need to 'make' again.

output is firmware-combined.bin located in esp8266/build

This is the current content of my modules dir:
 1624 Sep 15 16:39 websocket_helper.py
 2748 Sep 15 16:39 webrepl_setup.py
 2118 Sep 15 16:39 webrepl.py
   31 Sep 15 16:39 upip_utarfile.py -> ../../../tools/upip_utarfile.py
   22 Sep 15 16:39 upip.py -> ../../../tools/upip.py
 1094 Sep 15 16:39 port_diag.py
   35 Sep 15 16:39 onewire.py -> ../../../drivers/onewire/onewire.py
  850 Sep 15 16:39 ntptime.py
  836 Sep 15 16:39 neopixel.py
 1425 Sep 15 16:39 inisetup.py
 1110 Sep 15 16:39 flashbdev.py
   35 Sep 15 16:39 ds18x20.py -> ../../../drivers/onewire/ds18x20.py
   27 Sep 15 16:39 dht.py -> ../../../drivers/dht/dht.py
  219 Sep 15 16:39 _boot.py
  470 Sep 15 16:39 apa102.py
 1716 Sep 15 17:42 config.py
 6397 Sep 15 17:42 freesans34_num.py
 4096 Sep 15 17:42 uasyncio
   35 Sep 15 17:57 ssd1306.py -> ../../../drivers/display/ssd1306.py

Installing modules

Python uses a package manager to install modules, and it is also available for micropython. I'm not very familiar with all this, and I find it a bit fiddly. However, here are some instructions to install asyncio, logging, and picoweb.

Build micropython for unix.
cd into ports/unix, and here is a version of micropython which can run on the host. This is used to install modules locally before freezing them.

#first some prerequisite packages to install :
sudo apt-get install pkg-config python-pip libffi-dev

#next make the unix version of micropython
make axtls
make

#now you can download and install modules
#cd back into ports/esp8266
../unix/micropython -m upip install -p app picoweb (also installs uasyncio)

../unix/micropython -m upip install -p app micropython-logging

now I move all the files/dirs from app to modules - I suppose you could just install to modules first, but I liked the buffer of not messing up my modules directory, but instead checking the app directory before deciding what to move.

After that just make to build your new firmware, and you're good to go.

Thermostat in progress

As a potential upgrade for my Raspberry Pi heating system, I am working on a thermostat module for it - still in the prototype stages, more info to come but here's where we're at now:

eGalax 7" screen raspberry pi with sketch + GUI

I wanted a light installation of raspian, so I got sketch and added appropriate stuff to make x work.

To install GUI
sudo apt-get install xserver-xorg lxde lightdm

This stuff was required to make the touchscreen work - otherwise the calibration just didnt do anything.

sudo apt-get install libx11-dev libxext-dev libxi-dev x11proto-input-dev xserver-xorg-input-evdev xinput-calibrator  evtest -y


cd /usr/share/X11/xorg.conf.d
sudo mv 10-evdev.conf 45-evdev.conf


sudo vi 99-calibration.conf
add xinput_calibrator output, which should look like this:


Section "InputClass"
        Identifier      "calibration"
        MatchProduct    "eGalax Inc. USB TouchController"
        Option  "Calibration"   "67 1948 1877 183"
        Option  "SwapAxes"      "0"
EndSection

Broken Ribs

Three weeks ago today I had a kite-surfing accident and broke/fractured rib(s). At the time it was uncomfortable, but I carried on surfing for a few minutes, and it was time to pack up anyway.

I was able to pack up, get home, unpack the car, get out of my wet-suit etc, all a little uncomfortable, but really not too painful. The next morning I went to drive to work, and I got about 100 yards in the car and the clicking and pain was almost unbearable. It still makes me a bit light-headed to think about it.

I limped home, and ended up at home for 3 weeks.

The first week I really did no activities at all. I wasn't away from the house for the first week.

I got some stronger painkillers on prescription, and began taking them immediately. I struggled to do basic things like getting up the stairs, or standing and sitting. All week I lay exactly flat on my back all night without moving. Very uncomfortable.

By the time one week had passed, I was able to be a passenger in a car. I was able to get in and out of a car with a bit of care. However, at this stage I was struggling to get in and out of bed, I made sure I only made the transition from horizontal to vertical once per day. No laying about on the sofa, I only spent time sitting straight up during the day.

After about 10 days I tried driving a short distance, but it was still very uncomfortable, so I gave up driving for that second weekend. Each day I felt more comfortable, and at this stage I was able to sleep a bit on my side; my good side only, not the injured side.

Today, 3 weeks after my injury I returned to work. I'm still clicking a bit, and to be honest, driving is probably the activity I find most difficult. I'm much more comfortable sleeping, in fact last night was the first night I got to sleep on my injured side.

I still have some way to go, but I'm feeling incredibly better than I was, and each day brings an improvement. It wont be long before I'm strapped to a kite again ;)

Kitesurfing - Downhill - 11th September 2017

Epic conditions tonight at Downhill, nice wind for my new 9 and the longest flats between the waves I've ever seen. Out after work, on the water for about 90 minutes.

Then I tried a down loop waterstart. Epic fail, crashed kite and face plant. Sore, but quickly recovered. Then I tried a back roll, and I really have no idea what happened, but I fell hard on my left side.  Real hard. So hard in fact that I think I have broken more than one rib. They are cracking and popping like mad. 

I suspect I'll be out of the water for a week or two :(

Kitesurfing Benone 9th September 2017

Today was forecast to be good north west all day, but better in the afternoon with more sunshine. I went to Benone after lunchtime and there were 4 other kite surfers already setting up. They were strangers to me, seemed nice, and also mostly learners, I think.

I set up and set off, but it quickly became apparent that these guys were in my way. Being learners their kites were almost always at 12 o clock and I was upwind of them, meaning I couldn't easily sail past them. The wind was quite onshore too.

So I packed off down the beach, downwind of them. It was great. Great conditions, and the new kites boost a lot more than the old ones. Really enjoying them. I was there for a few hours, just getting to grips with the new kite, and how quick it is compared to my old ones. Great Saturday afternoon!!

Kitesurfing - Benone - Friday 8th September 2017

Forecast looked good for this evening all week, and it actually happened. I don't think I have been this nervous about kitesurfing for ages, taking to the water on a new kite, and not knowing its characteristics.

Anyway, after work I headed straight to the beach, someone else wa kiting, so I went to near them. I took a lot longer than normal to get set up, as I was making sure everything was ok with the gear.

It was quite showery, but between the showers I had a great wee session. This is the first time I easily stayed upwind in a long time. Totally juiced on the new 9.

Kite 5th September

Tonight I went to the beach to try out my new kites. They arrived in the post today, two Ozone C4 kites, 9 and 12.

I was really only hoping to set them up and see what they flew like, but unfortunately there really wasn't enough wind to even do that. Maybe another day.

Kitesurfing 26th Jun 2017

Great wee session this afternoon down at Downhill. Wind was quite changeable, I tried both the 9 and the 12, but settled on the 9. Ended up doing two downwinders from the river to the mouth. Lovely flats between the waves, and nice ramps, up to 6-8 ft at the outside. Cross from the west side. Best day I've had for ages. The 9 is so nimble, makes me wonder if I should change the 12....

Connecting to a Raspberry Pi behind a 3G Router

Experimenting with 3G router and '3' free 200MB/Mth data SIM and of course there is the NAT problem, so I did this:

http://zieren.de/raspberry-pi/reverse-ssh-through-3gnat/

/usr/bin/sshpass -p "password" ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=15 -N -R 22222:localhost:22 pi@my-house.ddns.net -p 1234

Then I can "ssh localhost -p 22222" on my local pi and get connected to the one behind the NAT. Excellent.

Kitesurfing 5th May 2017

Roasting weather, east wind, great!

Raspberry pi eGalax touchscreen calibration on zero / Raspberry pi 3 / PIXEL

Software updates bring compatibility problems, that's just how it is. Although the eGalax touch screen drivers are now in the kernel, and the touch screen functionality works out of the box, I couldn't get the touch calibrated.

Long story short, this is what I ended up with. Sometime xinput_calibrator came out with a section like this :

 Section "InputClass"
        Identifier      "calibration"
        MatchProduct    "eGalax Inc. USB TouchController"
        Option  "MinX"  "2881"
        Option  "MaxX"  "62736"
        Option  "MinY"  "60381"
        Option  "MaxY"  "5495"
        Option  "SwapXY"        "0" # unless it was already set to 1
        Option  "InvertX"       "0"  # unless it was already set
        Option  "InvertY"       "0"  # unless it was already set
 EndSection

but that was no good!

Then, finally, it came out with a section like this:

 Section "InputClass"

        Identifier      "calibration"

        MatchProduct    "eGalax Inc. USB TouchController"

        Option  "Calibration"   "103 1962 1864 158"

        Option  "SwapAxes"      "0"

 EndSection

with the "Option Calibration stuff on one line, and this seemed to work. I put it in /usr/share/X11/xorg.conf.d/99-calibration.conf

Hope that saves someone else the time :)

moving from inittab to systemd

For better or for worse raspbian has moved from the old System V init process control to the newer systemd version. I liked my inittab file, because it was quite simple and extremely easy to use, but it is no longer a choice.

So, I had this old line in inittab:
Gar:2345:respawn:start-stop-daemon --start -p /tmp/garage.pid  -m --exec /usr/bin/java  -v -d /home/pi/builds -- -jar garageController.jar

And now I want to run it on a new Raspberry Pi.

So, we need to create a service, its actually quite simple. I created the file /etc/systemd/system/garage.service with this text:
[Unit]
Description=Garage Door Control Software

[Service]
Type=simple
PIDFile=/var/tmp/garage.pid
ExecStart=/usr/bin/java -jar garageDoor.jar gpio
#ExecStart=/home/pi/builds/start_controller.sh
WorkingDirectory=/home/pi/builds
Restart=always

[Install]
WantedBy=multi-user.target

Then we need to run a few commands to make sure our new service auto-starts.

sudo systemctl daemon-reload

sudo systemctl enable garage.service
sudo systemctl start garage.service

sudo systemctl --all | grep garage

##this should show that the service is started.

That's it. 

(I used instructions from https://n3mesisfixx.blogspot.ie/2012/08/migrate-etcinittab-job-to-systemd-in.html)

Raspberry pi wont update time on NTP

Weird problem with a pi zero-w, where when I put it on my friend's network it gets the wrong time.


Bit of a workaround, rather than a fix, but it got me up and running, and should be reliable...

1. Open rc.local file
   sudo nano /etc/rc.local
2. add following line to the rc.local file
   sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
3. reboot the Rpi
   sudo reboot

Basically, it gets the HTTP headers from the server at google.com and sets the pi's time to that.




From:
https://raspberrypi.stackexchange.com/questions/47542/raspberry-pi-wont-update-time

Using a Raspberry Pi to fix a broken Ethernet mains switch

Energenie LAN Switcher

My father-in-law uses an Energenie LAN switch to remotely manage the lighting in a shed where animals are kept. This means that the animals can be monitored remotely day and night by CCTV, and the lighting can be controlled either via the local web server on the LAN Switch, or via Energenie's internet web interface.

However, it stopped working. Time to call in the Raspberry Pi.

I already control my central heating using a Raspberry Pi and an off-the-shelf relay module, so I wondered if the exact same solution would work for the shed lights. I also wondered if I could shoehorn a Pi and the relay module into the energenie casing, so I opened it up.

Inside with the old electronics - you can see scorch marks from the old board failing

It looked like there was enough space there, so I removed the old control board with the burnt out components, and the next stage was to get the software and hardware working. I had an 8 channel relay board and an old Raspberry Pi lying around, so I decided to use those.

It is a while since I installed or updated Raspbian, so I downloaded the latest version, and I have to say, it seems that there is a lot more there by default now, which is helpful. I quickly got my heating server installed and configured, and connected the relay board, and it all worked easily.

The Pi with a relay module - I'm only using 4 relays

Using this I was able to connect via the PHP web interface, or via the android app I wrote for it. It's strangely satisfying to hear relays click on and off when you press buttons on your phone.

As this is really just a quick fix I wasn't going to change the app at all, so it still gets the title "heating controls" and the "on" command is "boost" like it is for the heating.

I set the boost to put the lights on for 6 hours, so that is effectively as long as would be required in one go. The heating controller backend has no concept of "on" or "off", only timed programs.


With this all working, it was time to box it all up. This was probably the trickiest part of the project, but I managed to get the relay module and Pi into the box. I had to do a bit of filing, but it turned out to fit quite snugly allowing the Ethernet connection out the end for easy access.

	Relay board wired in
	Everything fitting snugly

And some photos with the cover back on

The system has been in the shed for a few weeks now, and is working perfectly. The app allows connection over the local LAN or over the internet seamlessly, depending on which network is available, and the text based protocol means that with even with only GPRS or Edge the switching is very fast.

ATEM Television Studio - live copies of the output file

We recently installed a ATEM Television Studio to mix cameras to an MP4 file in our church building. The idea originally was that we would burn this to DVD, and augment our CD ministry to include video distribution on DVD.

ATEM Television Studio

Since the days when we used cassette tapes, we have always distributed the media directly after the service - so people waiting 5 minutes or so afterwards could take copies of the service to deliver to those who can't make it out.

We wanted to keep this up, and so we looked into a DVD recorder to record the output of the Television Studio module directly to DVD, which could then be duplicated after the service. However, we hit several technical hurdles with this, and had to give up - mainly around aspect ratio.

So we were stuck with the MP4 file on disk. The process of converting this to DVD video, and burning it is slow. Even just burning the 2-3GB MP4 file directly to a DVD for duplicating is a slow process.

After a lot of head scratching we decided to distribute copies of our services on USB memory stick. Many people have a modern TV with a USB input, and we were surprised how many of our elderly people actually had such a device. With this, it is relatively easy to play the video file in superb quality on a TV.

But, we are still left with a problem - how do we write ~3GB of data to multiple USB sticks after the service? Writing to a stick could take 11 minutes, or about 3 minutes for USB 3.0 - but even at 3 minutes, if we have multiple copies to produce it could take quite a while.

Again, more head scratching, investigations into hugely expensive USB duplicators, and USB 3 hubs, but no satisfactory solution. Then the idea struck - why don't we write the data to the USB stick while the service is running - so we tail the file as it grows on the PC, and write the data out to each pen. The bandwidth, even in USB 2 would be more than sufficient to make quite a few simultaneous copies.

So I set to work on a piece of software to do this. I got over the problem of reading from a locked file quite easily, and had a prototype. However, when I ran it, the copy wouldn't play. By looking at binary diffs of the files I discovered that MP4 files have header information at the beginning, which is actually written at the very end of the process. I also discovered the ATEM seems to write large portions of the file filled with zeros, and then fills up the zeros with actual data.

Long story short, after many iterations of my software I got it working. Two strategies helped me:

1. I don't read to the end of the file until it has been completely written - I stay about 5MB back from the end - this way I don't read any zeros.
2. As I read each chunk and write it out to the USB sticks, I keep the offset, size and checksum of the data in an list. After the writing is complete, I re-read the source file from the start, and recalculate the checksum. If the new sum is different, I seek to the stored location in the file and rewrite the specified chunk. Typically it only rewrites the first chunk.

I added an option into the program to check the checksum, but since it is working properly, there is now no need to read the entire written data from the stick. 

Copies of the service are available about a minute after the service ends. Actually, these are available long before the CDs of the service become available now! And of course, the same software can be used to make copies of any service, not just the one we just recorded. It means you can make many unattended copies, only limited by the number of ports you have.

Why not YouTube?

Some people have suggested distributing service content via YouTube, but I'm not sure that fits with what we are trying to do: 

1. It would take a long time to upload and process - some of our elderly people are watching their copy about an hour after the end of the service
2. We could put internet access into the church building and stream live
1. that would have associated ongoing costs
2. our target audience for this ministry generally don't have easy access to the internet
3. publishing online brings other concerns about who can see the content
3. Each USB stick is brought into the home of someone who may be housebound, and not see very many visitors during the week. This contact is costly (time/effort) for the person who delivers the media, but incredibly valuable to the person who receives it..

YouTube may be something we think about for the future, but we don't see it as a suitable extension of our current ministry.

Screen Blanking On Raspberry Pi With Official Touch Screen

On the official RPi touch screen the screen goes blank after 30 minutes, and at first glance this seemed to do what I spent a lot of time scripting for an HDMI screen.

However, I noticed that using this default screen blanking the touch which unblanks the screen also gets sent to whatever program is below it, so if you happen to touch a button on the 'blank' screen, the button gets a press event.

So, in order to fix this I installed the xscreensaver

sudo apt-get install xscreensaver

and then configure it using

xscreensaver-demo

which allows me to set it to a black screen, and I also noticed there is the option there to have it activate the power management of the screen to turn it off. For this application I want the screen to go off after 1 minute, so I set that, and very easily I have the functionality I need.

The screen goes off after 1 minute, and any touch instantly brings the screen on, but doesn't get passed through to the underlying application. Perfect.

I also removed the default screen blanking, as I suspect there is a bit of an issue sometimes

edit /etc/lightdm/lightdm.conf

[SeatDefaults]
xserver-command=X -s 0 -dpms

 Credit: http://www.geeks3d.com/hacklab/20160108/how-to-disable-the-blank-screen-on-raspberry-pi-raspbian/

"O death, where is thy sting?" I think I found it.

I don't mean to be disrespectful in my treatment of scripture, but this text "O death, where is thy sting?" (1 Corinthians 15:55) has been on my mind over the last couple of days, and more-so today as we attended the thanksgiving service for a dear friend, and a devoted follower of Jesus Christ.

Since I heard the news on Sunday, when I walk, as I drive, as I go about life, I think to myself, boy, this stings. As we sang part of Psalm 116 on Sunday "Dear in God's sight is his saints' death" it stings. As I think of children and grandchildren without a mum and granny, it stings. As I think of a husband without a wife....

Death stings because we are the ones who are left behind. 

Ann was full of life. Ann was full of "the life" - Jesus said "I am the way, the truth and the life" - this is the life that Ann had living inside of her. And it was easy to see, and it stood the test of time and tribulation. Jesus said "the one who stands firm to the end will be saved"  (Matthew 24:13) meaning that one of the tests of our faith is time. There would be those who would initially spring up very quickly, and look like they had faith in Jesus, but they would quickly wither and it would be clear that their faith had no real roots (Matthew 13:20-21). Ann was the opposite of this. As troubles and difficulties multiplied, so did her faith. As trouble comes it provides a real, visible test of our faith, demonstrating if we are true or false converts. 1 Peter 1:6-7 makes it clear that trouble will come, and that this trouble will ensure that our faith "may be proved genuine and may result in praise, glory and honour when Jesus Christ is revealed." This is what happened. As Ann lived more vibrantly than ever in her last year, the glory of God was shown clearly. As her illness increased her health declined yet her hope and peace increased. She had "no guilt in life, no fear in death" - a wonderful testimony to all who looked on.

“Where, O death, is your victory?
    Where, O death, is your sting?”

The ultimate victory of death is gone. The ultimate sting of death is gone. Guilt from sin is removed for those who repent and trust. We can look at death and say "to live is Christ, to die is gain" - how is it not gain to fall asleep, and wake up in the direct presence of the Saviour God? How great it will be to find relief from the troubles of this world, in the place where "There will be no more death or mourning or crying or pain, for the old order of things has passed away." (Revelation 21:4)

Ann has fallen asleep. She has gone to be with her King. For now we are separated. But the sad truth is that for many who attended the service yesterday that separation will permanent, there is no denying that, because Ann chose a way of life that many reject. Jesus said that the way was narrow, and few find it. He said that many "religious" people would say "Lord, Lord" to him on judgement day, and he will tell them he didn't know them. I want to be one who instead hears the words "well done, good and faithful servant" (Matthew 25). One day I want to stand beside her in the presence of Jesus; her saviour and mine.

As we who remain deal with this horrible sting that we feel, we need to remember that "God is our refuge and strength, a very present help in trouble" (Psalm 46:1).

Let each of us who remain look to Him as our fortress, our deliverer, our God, our rock, in whom we take refuge, our shield. Ann would want us to honour her memory by honouring the one who gave her life, and life eternal. She would want us to do that by finding our hope in the same place she had hers. Let us do her memory justice by placing our hope in the same God as Ann did.

“Let not your hearts be troubled. Believe in God; believe also in me. In my Father's house are many rooms. If it were not so, would I have told you that I go to prepare a place for you? And if I go and prepare a place for you, I will come again and will take you to myself, that where I am you may be also. (John 14:1-3 (ESV))

Soli Deo gloria

Talktalk HG633 Router Saga Summary

Just in case this helps anyone else diagnose their router:

If you have Fibre Broadband from Talktalk and you have a good reliable DSL connection to the exchange (no resyncs, no DSL drops), but your internet connection speed drops intermittently to 1Mbps or lower, and a reboot of the router normally fixes it, then you need a new router  - and not a Talktalk one.

My router had the latest firmware (according to their website, although one of the technicians said it was out of date). The network inside the house including wireless worked perfectly all the time.

My sync speed to the exchange was always good, and the DSL connection only dropped when I rebooted the router. Rebooting the router instantly fixed the problem, but it would return sometimes within minutes, sometimes many hours later.

They sent a BT engineer, who confirmed that the line was all good.

I also found out that simply restarting the DSL fixed the problem too. (Click Maintain then scroll down to DSL, then there is a restart DSL button right at the bottom).

After a 2 month diagnosis with Talktalk ended up getting a new Talktalk router and it had the same issues. I ordered a Netgear Nighthawk D7000 router and this solved my problems. Buy one from Amazon UK, and then you should be able to return it for a refund if it doesn't solve your problem (just don't take all the plastic off it etc)