CR or LF on a control page question

Posted on
Mon Aug 14, 2017 5:08 pm
norcoscia offline
User avatar
Posts: 1206
Joined: Sep 09, 2015

Re: CR or LF on a control page question

Wow, Terry - it worked - Thanks !!! - I plan to tackle (try to tackle) it again tomorrow - hope I can get it to work, fingers crossed....

Dave, that looks so good - I can only imagine how long it must have taken to get everything just right on your page. I have been working on a security control page for days and it is only a small notch above embarrassing.

You guys both rock!!!!

_______
Norm

Posted on
Mon Aug 14, 2017 5:10 pm
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

Happy to help and glad it worked for you. Give us another shout if you have more questions, and remember to have fun!

Terry

Posted on
Mon Aug 14, 2017 7:01 pm
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: CR or LF on a control page question

norcoscia wrote:
Dave, that looks so good - I can only imagine how long it must have taken to get everything just right on your page.

It may seem like a lot of work, but it's really not -- once you get a handle on the process. Both graphs are built almost completely using defaults in the Matplotlib plugin (which relies on the 24-hour and 10-day forecast devices). Then it's just a matter of choosing your favorite colors. The images are saved to Indigo's devices directory with names like weather+cloudy.png, weather+fog.png, etc. -- tied to the same WU devices -- and a few text elements thrown in.

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Posted on
Tue Aug 15, 2017 9:25 am
norcoscia offline
User avatar
Posts: 1206
Joined: Sep 09, 2015

Re: CR or LF on a control page question

roussell wrote:
This should get you really close. You'll need to substitute your location; I got the link below by clicking on "Full Forecast" from the main WU website, then clicking on the "Forecast" tab on the next page. Modify and save the following code somewhere (I used the filename "wufc.html" in this example):

Code: Select all
<!DOCTYPE html>
<html>
<head>
    <style>
        #timeline {
   width: 800px;
   height: 1351px;
   background-color:transparent;
   border: 0px none;
   margin-top: -970px;
   margin-left: -10px;
}
    </style>
    <script src="/scripts/snippet-javascript-console.min.js?v=1"></script>
    </head>
<body>
    <iframe id="timeline" src="https://www.wunderground.com/us/al/birmingham/zmw:35201.1.99999#div.plots.has-header" scrolling="no" frameborder="0" ></iframe>
    <script type="text/javascript">
       
    </script>
</body>
</html>


Then run webkit2png against it like so:
Code: Select all
webkit2png -C --clipwidth=800 --clipheight=400 --scale=1 file:///Users/terry/Desktop/wufc.html --transparent --delay=5 -o wu10day
to produce the following image:
wu10day-clipped.png


I had to use a 5 second delay to allow WU to build the forecast. Also note that this will work until they change something on their site. How this gives you a good starting point.

Terry


Hi Terry, sorry for what I'm sure is a dumb question but when you say - save the code - do you mean as an HTML file on your HA PC - and when you say "run webkit2png against it" do you just put the webkit line in an indigo schedule as a python script - when I do that - it does not like it - and when I run from the terminal window I just get a picture of the HTML text like it is a snapshot of the text pad text. - I think I need some help - if you don't have time no worries - I can always just look outside :shock:

_______
Norm

Posted on
Tue Aug 15, 2017 10:42 am
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

norcoscia wrote:
Hi Terry, sorry for what I'm sure is a dumb question but when you say - save the code - do you mean as an HTML file on your HA PC - and when you say "run webkit2png against it" do you just put the webkit line in an indigo schedule as a python script - when I do that - it does not like it - and when I run from the terminal window I just get a picture of the HTML text like it is a snapshot of the text pad text. - I think I need some help - if you don't have time no worries - I can always just look outside :shock:


Yep, take that come and save it something.html on your PC, it doesn't have to be your HA PC at this point, since we're just playing around. Use a good text editor like Text Wrangler, Sublime Text, or any other you're familiar with, just not the Apple-provided TextEdit, it does strange things to pure text files. If you're playing on a Windows PC, notepad.exe is fine - BUT - I'm not sure how the webpages will render on Windows, probably best to stick with a Mac/Safari for experimentation.

Anyway, once you have it saved as a plain text (html) file on your Mac, and named something.html, double-click on it so that it opens in Safari (assuming Safari is your default web browser). You should see some ads, or other strange garbage text flash up for a few seconds while the page loads, and then you should see the 10-day forecast. (That few seconds of 'garbage' is why we delay webkit2png 5 seconds before taking screenshot.)

Now assuming that the result of the above is the 10 day forecast for Birmingham, Alabama, as in the image I linked earlier - that part is working. You just need to change the location. Thats done in the line:
Code: Select all
https://www.wunderground.com/us/al/birmingham/zmw:35201.1.99999#div.plots.has-header
in the html file. Edit the HTML file and change that line (everything after the https://, and before the #div.plots.has-header to reflect your area. As I mentioned before, I got that URL by clicking on "Full Forecast" from the main WU website, then clicking on the "Forecast" tab on the next page. Look at the URL that Safari is and copy/paste that into the HTML file, replacing my Birmingham info. It's important to leave the #div.plots.has-header piece in there though, thats the HTML DIV that actually holds the 10 Day forecast on WU's webpage.

Save your changes and open it in Safari again. This time you should see your local info instead of mine. Assuming that works, you can move on to creating a png image from that webpage. Thats where webkit2png comes in. You run it from a terminal and point it at the html file you just created. So, the following command typed at a terminal window:
Code: Select all
webkit2png -C --clipwidth=800 --clipheight=400 --scale=1 file:///Users/terry/Desktop/wufc.html --transparent --delay=5 -o wu10day
is broken down like this:

webkit2png -- the command
-C -- This switch tells webkit2png to produced a "clipped" png, to the specifications we'll specify
--clipwidth=800 --clipheight=400 -- the size to crop the image. Note that this is a crop, and not a scale.
--scale=1 -- one-to-one image scale, adjust this is you want to zoom/shrink the resultant png image.
file:///Users/terry/Desktop/wufc.html -- the path to the html file you created, adjust the path & file to fit your situation
--transparent -- IF POSSIBLE, webkit2png will create the image with a transparent background. This isn't always possible and depends on how the donor webpage was coded.
--delay=5 -- Load the page, but wait 5 seconds before creating the png. This give the webpage time to load, change this depending on what you're grabbing, it'll take some trial-and-error
-o wu10day -- output png name. This tell webkit2png to create a file named wu10day. NOTE: it will append "-clipped.png" to the name, resulting in "wu10day-clipped.png"

If you have the HTLP file producing a satisfactory webpage in the first step, and if you adjust the parameters in the webkit2png command above to fit your situation, you should then have a nice PNG in the current directory from where you're running webkit2png. Assuming all of that works, you're ready to transfer the process to Indigo.

Move the html file to wherever you're storing your Indigo scripts. mine are located in /Library/Application Support/Perceptive Automation/Indigo 7/Scripts/", but you can have it pretty much anywhere it's convenient for you and Indigo can "see" it.

Now, I like to specify absolute paths to programs and files, that tends to make for long command when files are buried deeply. So mu usual practice is to consolidate similar commands into a shell script thats eaiser to reference. As an example, the command that generates my weather image looks like this due to the long absolute paths:
Code: Select all
/usr/local/bin/webkit2png -F /Library/Application\ Support/Perceptive\ Automation/Indigo\ 7/Scripts/forecast2.html --transparent -D /Library/Application\ Support/Perceptive\ Automation/Indigo\ 7/IndigoWebServer/static/ -o forecast --delay=3


So, what I did, and recommend you do as well, is in the same folder that you place your html file, also create a text file called getweather.sh or similar (just make sure to have the ".sh" on there.) and in that .sh text file, place two lines. the first line should be:
Code: Select all
#!/bin/sh
which is standard in a shell script and simply tells the command processor where to find the UNIX shell. The second line should be you're webkit2png command that you created above. Just up-arrow in the terminal and copy/paste it into the file. AS I did in my long webkit2png command above, be sure to type out the full path to the html, file, the directory where your storing the file (the -D switch in my command above) and the path to webkit2png itself (probably usr/local/bin/webkit2png). Save that and then back at the terminal, change to the directory where you have the script stored and type
Code: Select all
chmod a+x ./filename.sh
where filename.sh is the shell script you just created. IF that doesn't return an error, at the terminal type ./filename.sh and webkit2png should run with all of your paths and switches, producing another image after a few seconds.

Assuming all of that works, all you need to do now is create a schedule in Indigo to run that shell command at whatever frequency you desire. Here's mine:
Screen Shot 2017-08-15 at 11.38.28 AM.png
Screen Shot 2017-08-15 at 11.38.28 AM.png (215.61 KiB) Viewed 3542 times

Screen Shot 2017-08-15 at 11.40.40 AM.png
Screen Shot 2017-08-15 at 11.40.40 AM.png (272.64 KiB) Viewed 3542 times


Hope this helps, I know it seems like a lot, but just break it down into the individual pieces and you'll get it!

Terry

Posted on
Tue Aug 15, 2017 10:47 am
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

This would actually make a decent plugin, as I believe there is a web2png for Python, but work has me stretched thin at the moment, plus I've always been sort of a script-guy.. I rather enjoy having my directory of scripts for all sorts of stuff and just having Indigo orchestrate their execution. Perhaps one day, when I don't have one daughter starting college, and another daughter starting kindergarten, I'll have the time to tackle something for me. :!: :| :lol: :shock: (there really needs to be a sobbing uncontrollably emoji)

Terry

Posted on
Tue Aug 15, 2017 11:03 am
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

One other worthwhile mention - I remember seeing posts about a plugin that creates images from webpages. I don't remember the name, or the author, but it might be something worth exploring, if your not read to tackle all of the above. Perhaps somebody else reading along knows what plugin I'm thinking of.

Terry

Posted on
Tue Aug 15, 2017 11:12 am
jay (support) offline
Site Admin
User avatar
Posts: 18199
Joined: Mar 19, 2008
Location: Austin, Texas

Re: CR or LF on a control page question

roussell wrote:
(there really needs to be a sobbing uncontrollably emoji)


Image

There ya go...

Jay (Indigo Support)
Twitter | Facebook | LinkedIn

Posted on
Tue Aug 15, 2017 2:15 pm
norcoscia offline
User avatar
Posts: 1206
Joined: Sep 09, 2015

Re: CR or LF on a control page question

Thanks Terry - I'm working my way down your instructions - understanding everything so far but I have not set it up in Indigo yet - I do have one problem. The image that is produced does not seem to be in the right spot (or the 10 day forecast) see image below - I'm using

https://www.wunderground.com/us/wa/ridg ... 42.1.99999

for my location - wondering if our monitors have a different resolutions and that is somehow messing it up. But what I don't understand is when I use the info above to open a page the page seem correct but the grab looks like the image below????
Attachments
2017-08-15_12-24-45.png
2017-08-15_12-24-45.png (97.01 KiB) Viewed 3519 times

_______
Norm

Posted on
Tue Aug 15, 2017 5:06 pm
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

Try this for your HTML file:
Code: Select all
<!DOCTYPE html>
<html>
<head>
    <style>
        #timeline {
   width: 800px;
   height: 394px;
   background-color:transparent;
   border: none;
   margin-top: -10px;
   margin-left: -10px;
}
    </style>
    </head>
<body>
    <iframe id="timeline" src="https://www.wunderground.com/us/wa/ridgefield/zmw:98642.1.99999#forecast-graph" scrolling="no" frameborder="0" ></iframe>
</body>
</html>
I changed the DIV that is shown in the iframe, and changed the window size to fit. It looks correct to me, but as always, your mileage may vary. :)

Also, I'm getting really erratic results form WU at the moment. You might want to increase the delay from 5 seconds to 6 or 7 when you do the webkit2png command (the --delay switch)

BTW, just so you know what's happening here - an IFRAME is an HTML element that allows you to embed a webpage inside another one. It has the ability to either show an entire webpage, or -as we're doing here- act as a viewport of-sorts to the secondary page; like looking through a window and only seeing part of the landscape. The stuff contained in the <style> tags at the top of the script define the parameters for the iframe, it's size and positioning. Then the <iframe> tag tells the web browser what to view, and whether to show scroll bars. the iframe tag has been around since the late 1990s I think, but got a bit of a bad rap a decade or so ago when people were overusing and improperly using it, still a pretty powerful tool though!

Terry
Last edited by roussell on Tue Aug 15, 2017 5:37 pm, edited 1 time in total.

Posted on
Tue Aug 15, 2017 5:30 pm
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

jay (support) wrote:
roussell wrote:
(there really needs to be a sobbing uncontrollably emoji)


Image

There ya go...


Nailed it!

Posted on
Wed Aug 16, 2017 7:31 am
norcoscia offline
User avatar
Posts: 1206
Joined: Sep 09, 2015

Re: CR or LF on a control page question

I don't think this will ever work reliably - looks like (I think) different adds take up different amounts of space on the page so the exact location of the "window" I want changes over time - even though it will not work I really appreciate all the time you took to help - take care....

Norm

_______
Norm

Posted on
Wed Aug 16, 2017 7:42 am
roussell offline
User avatar
Posts: 1108
Joined: Aug 18, 2008
Location: Alabama

Re: CR or LF on a control page question

Sorry it isn't working for you, Ads and other dynamic stuff definitely makes scraping webpages harder. I attempted to nail it down a bit by just specifying the DIV that holds the 10-day, but I guess it still wasn't enough. The one I use is from dark sky, a "widget" they provide to embed on other webpages so it's fairly static. Take a look at the WU and Matplotlib Plugin as Dave showed earlier. It will take a little more work to get setup, but should be much more reliable over time.

Terry

Posted on
Wed Aug 16, 2017 7:55 am
norcoscia offline
User avatar
Posts: 1206
Joined: Sep 09, 2015

Re: CR or LF on a control page question

My only hope is that Dave adds a 10 day forecast plug-in thingy - tonight I'll dream of a plug-in that lets me just define the size of the screen and just like that - BAM - a perfect 10 day forecast con my control page...

BTW, I already have both plug-ins and getting Matplotlib to do that is way way way beyond my capabilities - no worries since I'm sure Jay is doing the final testing of the "10-Day" forecast control page command before it is released in V7.1 :mrgreen:

_______
Norm

Posted on
Wed Aug 16, 2017 4:11 pm
DaveL17 offline
User avatar
Posts: 6742
Joined: Aug 20, 2013
Location: Chicago, IL, USA

Re: CR or LF on a control page question

I don't envision generating graphical elements with the WUnderground plugin, but you never know. :D

I came here to drink milk and kick ass....and I've just finished my milk.

[My Plugins] - [My Forums]

Who is online

Users browsing this forum: No registered users and 1 guest