Ramblings of a Madman General Ramblings About all things JavaScript and Firefox Developer Tools http://flailingmonkey.com Running ESLint in Atom for Mozilla Development Mike Ratcliffe <p>Due to some recent changes in the way that we use eslint to check that our coding style linting Mozilla source code in Atom has been broken for a month or two.</p> <p>I have recently spent some time working on Atom’s linter-eslint plugin making it possible to bring all of that linting goodness back to life!</p> <p>From the root of the project type:</p> <div class="highlight"><pre><code class="language-javascript"><span class="p">.</span><span class="o">/</span><span class="nx">mach</span> <span class="nx">eslint</span> <span class="o">--</span><span class="nx">setup</span></code></pre></div> <p>Install the linter-eslint package v.8.00 or above. Then go to the package settings and enable the following options:</p> <figure> <p><img src="/assets/images/Eslint-atom-settings.png" alt="Eslint Settings" /></p> <figcaption>Figure 1: Eslint Settings in Atom</figcaption> </figure> <p>Once done, you should see errors and warnings as shown in the screenshot below:</p> <figure> <p><img src="/assets/images/Eslint-atom.png" alt="Eslint in Atom" /></p> <figcaption>Figure 1: Eslint in Atom</figcaption> </figure> Tue, 18 Oct 2016 17:00:00 +0100 http://flailingmonkey.com/running-eslint-in-atom-for-mozilla-development/ http://flailingmonkey.com/running-eslint-in-atom-for-mozilla-development/ Installing VPN from the Linux Command Line Mike Ratcliffe <p>To stop people snooping on what we are doing on the internet or when restrictive governments prevent people from accessing popular websites you can use a VPN to protect your privacy and allow you to access the internet without restrictions.</p> <p>Most VPN providers provide simple step-by-step instructions to get VPN set up on most operating systems but many fail to give instructions about how to autostart a VPN in a non graphical environment.</p> <p>The following will work with the files provided by most VPN providers with a couple of small changes:</p> <p>Of course, the url that your provider uses may be different:</p> <figure> <div class="highlight"><pre><code class="language-powershell"><span class="c"># Get the Files from your Provider</span> <span class="p">$</span> <span class="n">cd</span> <span class="p">/</span><span class="n">tmp</span> <span class="p">$</span> <span class="n">wget</span> <span class="n">https</span><span class="err">:</span><span class="p">//</span><span class="n">www</span><span class="p">.</span><span class="n">privateinternetaccess</span><span class="p">.</span><span class="n">com</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span><span class="n">openvpn</span><span class="p">.</span><span class="n">zip</span> <span class="p">$</span> <span class="n">unzip</span> <span class="n">openvpn</span><span class="p">.</span><span class="n">zip</span> <span class="c"># Copy Certificates to /etc/openvpn</span> <span class="p">$</span> <span class="n">cp</span> <span class="n">ca</span><span class="p">.</span><span class="n">crt</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span> <span class="p">$</span> <span class="n">cp</span> <span class="n">crl</span><span class="p">.</span><span class="n">pem</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span> <span class="c"># Enable Autologin</span> <span class="p">$</span> <span class="n">nano</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span><span class="n">login</span><span class="p">.</span><span class="n">conf</span> <span class="c"># Edit file to contain your username and password on the first two lines:</span> <span class="n">your-privateinternetaccess-username</span> <span class="n">your-privateinternetaccess-password</span> <span class="p">&lt;</span><span class="n">ctrl</span><span class="p">&gt;&lt;</span><span class="n">x</span><span class="p">&gt;</span> <span class="n">to</span> <span class="n">exit</span> <span class="n">and</span> <span class="n">Y</span> <span class="n">to</span> <span class="n">save</span><span class="p">.</span> <span class="p">$</span> <span class="n">chmod</span> <span class="n">400</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span><span class="n">login</span><span class="p">.</span><span class="n">conf</span> <span class="c"># Point OpenVPN Towards the Right Files. The .ovpn files may</span> <span class="c"># not be by country so substitute them as appropriate.</span> <span class="p">$</span> <span class="n">nano</span> <span class="n">Country</span><span class="p">.</span><span class="n">ovpn</span> <span class="c"># Edit the file to prefix the pem and crt paths with:</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span> <span class="c"># Change the auth-user-pass line to read:</span> <span class="n">auth-user-pass</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span><span class="n">login</span><span class="p">.</span><span class="n">conf</span> <span class="p">&lt;</span><span class="n">ctrl</span><span class="p">&gt;&lt;</span><span class="n">x</span><span class="p">&gt;</span> <span class="n">to</span> <span class="n">exit</span> <span class="n">and</span> <span class="n">Y</span> <span class="n">to</span> <span class="n">save</span><span class="p">.</span> <span class="p">$</span> <span class="n">cp</span> <span class="n">Country</span><span class="p">.</span><span class="n">ovpn</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="n">openvpn</span><span class="p">/</span><span class="n">Country</span><span class="p">.</span><span class="n">conf</span> <span class="c"># Autostart the VPN. &quot;Country&quot; is the name of your .conf file without</span> <span class="c"># any extension so in my case Country.conf (see the previous steps).</span> <span class="p">$</span> <span class="n">nano</span> <span class="p">/</span><span class="n">etc</span><span class="p">/</span><span class="k">default</span><span class="p">/</span><span class="n">openvpn</span> <span class="c"># Edit the file so it contains:</span> <span class="n">AUTOSTART</span><span class="p">=</span><span class="s2">&quot;Country&quot;</span> <span class="c"># Check that Autostart is Working</span> <span class="p">$</span> <span class="n">reboot</span></code></pre></div> <figcaption>Installing VPN from the Linux command line</figcaption> </figure> <p>To check if the VPN is running open lynx and go to google.com. It should take you to the specific countries site.</p> <p>That is it, you are done!</p> Wed, 02 Sep 2015 20:55:56 +0100 http://flailingmonkey.com/installing-vpn-from-the-linux-command-line/ http://flailingmonkey.com/installing-vpn-from-the-linux-command-line/ View jQuery and jQuery Live events in Firefox DevTools Mike Ratcliffe <p>A little while back we <a href="/view-dom-events-in-firefox-developer-tools/">landed a feature</a> that made it possible to interact with events from the markup panel of Firefox developer tools.</p> <p>A lot of people were excited about this but we soon realized that jQuery wraps event handlers in a proxy making all handlers look like this:</p> <div class="highlight"><pre><code class="language-javascript"><span class="kd">function</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> <span class="nx">thisObject</span> <span class="o">||</span> <span class="k">this</span><span class="p">,</span> <span class="nx">arguments</span> <span class="p">);</span> <span class="p">};</span></code></pre></div> <p>Numerous facepalms later and we now have support for jQuery and jQuery Live events. This support will be available in <a href="http://nightly.mozilla.org/">tonight’s nightly</a>:</p> <figure> <p><img src="/assets/images/jquery-and-jquery-live-events.png" alt="jQuery and jQuery Live events" /></p> <figcaption>jQuery and jQuery Live Events</figcaption> </figure> <p>Because library support is important we have created an API that allows developers and library authors to create “event parsers” that allow our tools to make sense of their event systems.</p> <p>This API is priviledged code so to create an event parser you will need to create a simple Firefox extension.</p> <p>The API looks like this:</p> <div class="highlight"><pre><code class="language-javascript"><span class="kr">const</span> <span class="p">{</span><span class="nx">EventParsers</span><span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s2">&quot;devtools/toolkit/event-parsers&quot;</span><span class="p">);</span> <span class="kd">var</span> <span class="nx">parser</span> <span class="o">=</span> <span class="p">{</span> <span class="nx">id</span><span class="o">:</span> <span class="s2">&quot;myLibrary events&quot;</span><span class="p">,</span> <span class="c1">// Unique id</span> <span class="nx">hasListeners</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">node</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Hunt for node&#39;s listeners and return true as soon as one is</span> <span class="c1">// encountered.</span> <span class="p">},</span> <span class="nx">getListeners</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">node</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Hunt for node&#39;s listeners and return an array of objects</span> <span class="c1">// representing those listeners. Each object should look like this:</span> <span class="p">{</span> <span class="nx">type</span><span class="o">:</span> <span class="s2">&quot;click&quot;</span><span class="p">,</span> <span class="nx">handler</span><span class="o">:</span> <span class="kd">function</span> <span class="nx">clicked</span><span class="p">()</span> <span class="p">{...},</span> <span class="c1">// These tags will be displayed as attributes in the events popup.</span> <span class="nx">tags</span><span class="o">:</span> <span class="s2">&quot;jQuery,Live&quot;</span><span class="p">,</span> <span class="c1">// Hide or show fields</span> <span class="nx">hide</span><span class="o">:</span> <span class="p">{</span> <span class="kr">debugger</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="c1">// Debugger icon</span> <span class="nx">type</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="c1">// Event type e.g. click</span> <span class="nx">filename</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="nx">capturing</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="nx">dom0</span><span class="o">:</span> <span class="kc">false</span> <span class="p">},</span> <span class="nx">override</span><span class="o">:</span> <span class="p">{</span> <span class="c1">// The following can be overridden:</span> <span class="nx">type</span><span class="o">:</span> <span class="s2">&quot;click&quot;</span><span class="p">,</span> <span class="nx">origin</span><span class="o">:</span> <span class="s2">&quot;http://www.mozilla.com&quot;</span><span class="p">,</span> <span class="nx">searchString</span><span class="o">:</span> <span class="s1">&#39;onclick=&quot;doSomething()&quot;&#39;</span><span class="p">,</span> <span class="nx">DOM0</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">capturing</span><span class="o">:</span> <span class="kc">true</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">},</span> <span class="nx">normalizeHandler</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">fnDO</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Take a handler debug object and use the debugger to walk the scope</span> <span class="c1">// chain to discover the function you would like to be displayed.</span> <span class="c1">// See https://hg.mozilla.org/integration/fx-team/diff/9add1ec0251d/toolkit/devtools/event-parsers.js#l1.98 for an example.</span> <span class="p">}</span> <span class="nx">EventParsers</span><span class="p">.</span><span class="nx">registerEventParser</span><span class="p">(</span><span class="nx">parser</span><span class="p">);</span></code></pre></div> <p>We have bugs logged to add support for the following libraries:</p> <ul> <li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1044933">Prototype</a></li> <li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1044935">YUI</a></li> <li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1044936">Mootools</a></li> <li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1044939">React</a></li> <li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1044942">Ember</a></li> <li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1044944">Angular</a></li> </ul> <p>You still here? What are you waiting for? <a href="https://wiki.mozilla.org/DevTools/Hacking">Go get hacking</a>!</p> Thu, 28 Aug 2014 12:40:46 +0100 http://flailingmonkey.com/view-jquery-and-jquery-live-events-in-firefox-devtools/ http://flailingmonkey.com/view-jquery-and-jquery-live-events-in-firefox-devtools/ View DOM Events in Firefox Developer Tools Mike Ratcliffe <p>I recently realized that support for inspection of DOM events is very poor in pretty much all developer tools. Having seen Opera Dragonflies implementation some time ago I liked the way you could very easily see the scope of an event.</p> <p>I have used a similar design to add DOM event inspection to Firefox Developer Tools. The event icons are visible in the markup view and if you click on them you can see information about the event including it’s handler. If you click the debugger icon at the top left of the popup it will take you to that handler in the debugger.</p> <figure> <p><img src="/assets/images/visual-events-in-firefox-developer-tools.png" alt="Visual Events in Firefox Developer Tools" /></p> <figcaption>Visual Events in Firefox developer tools</figcaption> </figure> <p>Whilst developing this feature I noticed that my workflow changed considerably. I found myself repeatedly looking at the event handlers attached to e.g. a button, clicking the debug icon, adding a breakpoint and clicking the button.</p> <p>We hope this feature will be useful to you. If you have any idea how we can improve this feature then please let us know via <a href="http://ffdevtools.uservoice.com/forums/246087-firefox-developer-tools-ideas">our feedback channel</a> or <a href="https://bugzilla.mozilla.org/enter_bug.cgi?component=Developer%20Tools%3A%20Inspector&amp;product=Firefox&amp;rep_platform=All&amp;op_sys=All">Bugzilla</a>.</p> Mon, 21 Jul 2014 10:51:00 +0100 http://flailingmonkey.com/view-dom-events-in-firefox-developer-tools/ http://flailingmonkey.com/view-dom-events-in-firefox-developer-tools/ Hack your Family: Adopt a child (UK version) Mike Ratcliffe <p><img src="/assets/images/adoption.jpg" alt="Adoption" /></p> <p>A lot of people would like to adopt but do not know what to do or what is involved. They are often scared of the process or have heard horror stories. Being at the end of the process and having met our kids we can say that you should be completely open and honest with your social worker… even things that many would expect to be negatives are very strong positives when viewing you from a social worker’s perspective.</p> <p>e.g. If you were once a drug addict then you may have insights into the birth parent’s life if they suffered from the same issue. You would also be more capable of helping the child if they fall foul to drugs in their teenage years. Being honest will help the local authority match the correct child or children with your family.</p> <p>The process is difficult, partly because you will need to jump through all of the hoops that the adoption people ask you to jump through. Also, say you are adopting three kids around two years old… that means that you will need to buy everything that a birth parent would have bought over the first couple of years of each child’s life. This is different from giving birth where you tend to buy things as you need them but as an adoptive parent you will need to hit the ground running.</p> <p>Here are some of our preparations (by the letter we didn’t “need” all of these things but we are adopting three):</p> <ul> <li>Changed car to a model that supports three childseats across a single row (we need the boot / trunk for the buggy).</li> <li>Childseats for car.</li> <li>Changed downstairs carpet to laminate.</li> <li>Put toddler swing, slide and roundabout in back garden.</li> <li>Put childlocks on everything.</li> <li>Put safety gates everywhere.</li> <li>Redecorate two rooms as nurseries.</li> <li>Buy cot beds.</li> <li>Buy high chairs.</li> <li>Buy new fridge freezer.</li> <li>Buy baby monitor capable of monitoring two rooms.</li> <li>At least a million other things.</li> </ul> <p>Here is an outline of the process in the UK although it can vary slightly from county to county. The process is now supposed to be completed within six months due to the government making changes so if you really want kids you could have them within six months.</p> <h2 id="initial-enquiry">Initial Enquiry</h2> <p>You as a prospective adopter call your county council’s adoption team who talk to you about your motives and the adoption process itself. A social worker will visit you in an attempt to find any issues that would prevent adoption. Try not to take this visit too personally as they need to ensure that you really are serious about adopting.</p> <h2 id="registration--checks">Registration &amp; Checks</h2> <p>The adoption department will send an official application form that you will need to fill out.</p> <p>Another social worker will visit you lots of times asking probing questions. Do not underestimate how many visits there will be as the social worker will need to prepare a report about you that is so thick they often call it “The Book.” Some of the things they will ask about:</p> <ul> <li>Name</li> <li>Date of birth</li> <li>People living in the home</li> <li>Income</li> <li>Occupation</li> <li>Health</li> <li>Why do you want to adopt?</li> <li>Chronology (everywhere they have ever lived and worked including dates).</li> <li>Preferably every boss you have ever had will need to send a written reference about you to your social worker.</li> <li>Details of three referees who will need to complete a written reference. Two of these referees will need to be interviewed by your social worker.</li> <li>The type of child you want to adopt e.g. would you accept a child with schizophrenia, AIDS, ADHD etc.</li> <li>Medical background check.</li> <li>Criminal background check for every country you have lived in.</li> <li>Local authority checks for everywhere that you have lived.</li> </ul> <h2 id="preparation-groups">Preparation groups</h2> <p>There are many classes designed to train you for special situations that may arise with adopted children. Some examples are:</p> <ul> <li>Adoption Triangle: Learn about the child’s place in the adoption triangle (birth parents, adoptive parents and child).</li> <li>Can’t do, Won’t do – Attachment Difficulties for Children who have Experienced Early Trauma, Neglect and Abuse.</li> <li>Life Story Book (a kind of life journal that the child can use to find out about life before they were adopted and which will be updated regularly).</li> <li>‘Letterbox’ – Information Sharing between Adoptive Families and Birth Families.</li> <li>Emergency first aid and basic life support.</li> <li>Managing Behaviour for Under 10s.</li> <li>Separation and Loss.</li> </ul> <p>This is a small sample of available classes… there are many available. Some of these classes are core classes and must be attended. The rest are voluntary although some may be very suited to your adoptive child so you would be expected to attend them.</p> <h2 id="pre-assessment-decision">Pre-Assessment Decision</h2> <p>The social worker will meet with you. They will discuss the information gathered and confirm that you wish to proceed.</p> <h2 id="adoption-panel">Adoption Panel</h2> <p>Prospective adopters are sent the report that has been put together about them so far in the process. They are free to clarify any issues about the report with the social worker. This report is very detailed (about 60 - 70 A4 pages).</p> <p>Prospective adopters attend a panel (similar to a small court) where a group of experts ask questions to ensure that nothing has been missed in the process. This panel can give the adoption team advice on which children could be placed with the prospective adopters.</p> <p>After the panel the prospective adopters are told whether the panel accept their request to adopt (this is just an informal approval as the agency decision maker has the last say).</p> <p>Within two weeks the agency decision maker will review the panels decision and will send out a letter either approving or rejecting the prospective adopter’s adoption application.</p> <h2 id="being-matched-with-a-child">Being matched with a child</h2> <p>It is not usually as simple as a prospective adopter saying that they would like a particular child. Throughout the process the social workers will be discussing your case and looking out for children that would be a good match with your family.</p> <p>If no match is made at this point then prospective adopters first need to find out about available children and make a decision from there. There are a number of platforms available:</p> <ul> <li>Parents for children evenings: On these evenings prospective adopters are shown videos of children that are up for adoption. They are also shown booklets containing basic information about the children available including medical and behavioral difficulties etc.</li> <li>Activity days: Prospective adopters are given the opportunity to see a selection of available children. Many people do not like these activities, comparing them to cattle markets, but they are a great way to see the children in real life and get to know their personalities.</li> </ul> <p>Once you have found the child or children of your dreams you will be provided with a Child Permanency Report (<abbr title="Child Permanency Report">CPR</abbr>) which gives details about the child’s history and the events leading to them becoming available for adoption.This report may include information about medical issues, psychological issues and learning difficulties that the child may have.</p> <p>If no match has been made within three months prospective adopters can also be referred you to the National Register which maintains a list of approved adopters and children waiting for adoption placements to enable matches to be made nationally.</p> <h2 id="matching-panel">Matching Panel</h2> <p>A panel of experts will review your report and reports written about the children and discuss any concerns with you. This panel is like a small court.</p> <h2 id="adoption-placement-planning-meetings">Adoption placement planning meetings</h2> <p>Once matched with children a plan needs to be drawn up in order to make a gradual transition from the foster home to the home of the prospective adopters. This transition usually lasts between one and two weeks.</p> <h2 id="introductions">Introductions</h2> <p>This is where the plan from the previous section is put in place and you will spend time gradually getting to know the children and their routines at the foster parent’s home. This may require a lot of travel and takes a good couple of weeks.</p> <p>A good foster parent will gradually give you control of the children’s routines. In time the foster parents wil bring the children to visit you in your home where you will carry on with those routines. Eventually, they will go home and leave the children with you.</p> <p>This is a very emotionally exhausting time so both you and your partner will need to take some time off work.</p> <h2 id="placement-for-adoption--adoption-order">Placement for adoption &amp; adoption order</h2> <p>The child remains a ‘Looked After’ child until an Adoption Order is made. So until then the Local Authority shares Parental Responsibility with you, and will continue to hold reviews to discuss with you and the social workers how things are going and what support you and / or the children may need.</p> <p>After the child has lived with the prospective adopters for ten weeks, the law allows them to apply to the court for an Adoption Order.</p> <p>After the court makes the Adoption Order they will plan a “Celebration Hearing” where they meet the children and their new parents.</p> <h2 id="post-adoption--support">Post Adoption &amp; Support</h2> <ul> <li>Adoption support groups.</li> <li>Training events and workshops.</li> <li>Support and advice on contact with birth family including providing a “letterbox service” where indirect contact has been agreed with you regarding birth relatives.</li> </ul> Tue, 06 May 2014 10:47:00 +0100 http://flailingmonkey.com/hack-your-family-adopt-a-child/ http://flailingmonkey.com/hack-your-family-adopt-a-child/ Firefox Developer Tools Highlighter Mike Ratcliffe <p>When it comes to the Firefox Developer Tools Inspector one of our most common requests is that our highlighter should support box model highlighting. We are working on implementing this but before we get into full swing we need to know exactly what you want it to look like and how you would like it to act.</p> <p>Please take a look at how current tools behave and let us know what you would like our box model highlighting to look and behave.</p> <p>The examples on this page show the different methods of inspecting this div in Firefox Developer Tools, Firebug, Chrome DevTools and Opera Dragonfly:</p> <div style="width: -moz-fit-content; margin: 0px auto;"> <div style="width:220px; padding:10px; border: 10px dashed green; margin:10px; font: 14px/13px sans-serif;"> This is a div with: * padding: 10px; * border: 10px dashed green; * margin: 10px; </div> </div> <h2 id="inspection-by-mousing-over-content">Inspection by mousing over content</h2> <p>This is the mode that most people are familiar with. You click the inspect button and move the mouse over the content area to highlight a node.</p> <p>Firefox Developer Tools use a subtle outline around the border of the hovered element. In addition, a node infobar appears. This infobar allows you to set pseudoclasses such as :hover, delete the node, copy HTML etc. This is currently the only highlight mode used by the tools:</p> <figure> <p><img src="/assets/images/mouse-inspection-firefox.png" alt="Mouse Inspection (Firefox)" /></p> <figcaption>Mouse inspection Firefox</figcaption> </figure> <p>Firebug uses a blue outline:</p> <figure> <p><img src="/assets/images/mouse-inspection-firebug.png" alt="Mouse Inspection (Firebug)" /></p> <figcaption>Mouse inspection Firebug</figcaption> </figure> <p>Chrome DevTools highlights content, padding, border and margin areas even when inspecting via the mouse. They also display simple node information e.g. node name, width and height:</p> <figure> <p><img src="/assets/images/mouse-inspection-chrome.png" alt="Mouse Inspection (Chrome)" /></p> <figcaption>Mouse inspection Chrome</figcaption> </figure> <p>Opera Dragonfly also highlights content, padding, border and margin areas even when inspecting via the mouse. In addition it shows guides outlining the border of the element:</p> <figure> <p><img src="/assets/images/mouse-inspection-opera.png" alt="Mouse Inspection (Opera)" /></p> <figcaption>Mouse Inspection Opera</figcaption> </figure> <h2 id="inspection-by-mousing-over-the-layout-box-model-panel">Inspection by mousing over the layout (box model) panel</h2> <p>Firebug brings out the big guns here and highlights all of the different layout components. It also outlines the node’s active container in a dashed red line and adds rulers to it (useful for absolutely positioned elements). In case this is not enough it adds guides around the section being hovered (content, border etc.):</p> <figure> <p><img src="/assets/images/box-model-content-hover-firebug.png" alt="Box model content hover (Firebug)" /></p> <figcaption>Box model content hover Firebug</figcaption> </figure> <p>Chrome DevTools highlights only the content, padding, border or margin depending on which is being hovered:</p> <figure> <p><img src="/assets/images/box-model-content-hover-chrome.png" alt="Box model content hover (Chrome)" /></p> <figcaption>Box model content hover Chrome</figcaption> </figure> <h2 id="inspection-by-mousing-over-the-nodes-in-the-markup-html-panel">Inspection by mousing over the nodes in the Markup (HTML) panel</h2> <p>Firebug shows the box model (content, padding, border and margin):</p> <figure> <p><img src="/assets/images/html-panel-node-hover-firebug.png" alt="HTML Panel node hover (Firebug)" /></p> <figcaption>HTML Panel node hover Firebug</figcaption> </figure> <p>Chrome DevTools and Opera Dragonfly act exactly the same as when mousing over content.</p> <p>So the question is what would you like the Firefox Developer Tools highlighter to look like and how should it act in each of the situations above. It doesn’t need to act like any other highlighter so be creative!</p> Thu, 17 Oct 2013 10:42:00 +0100 http://flailingmonkey.com/firefox-developer-tools-highlighter/ http://flailingmonkey.com/firefox-developer-tools-highlighter/ Debugging Thunderbird using Firefox Developer Tools Mike Ratcliffe <p>I recently discovered that it is possible to use <a href="https://developer.mozilla.org/docs/Tools">Firefox Developer Tools</a> with <a href="http://www.mozilla.org/thunderbird/">Thunderbird</a>. <a href="https://twitter.com/pkewisch">Philipp Kewisch</a> has done a fantastic job of his Google Summer of Code 2013 Project to bring Firefox Developer Tools to Thunderbird.</p> <p>Starting with Thunderbird 24.0a1 and a matching version of <a href="https://www.mozilla.org/firefox/fx/#desktop">Firefox</a>, it is possible to debug Thunderbird code using Firefox Developer Tools. If your version of Thunderbird is 14.x then please go to <kbd>Help</kbd> → <kbd>About Thunderbird</kbd> and allow it to update.</p> <p>For best results you should use the latest nightly versions of both Thunderbird and Firefox. At the very least the version numbers should match. Here is how to get things up and running using the latest nightlies:</p> <ul> <li>Download the latest <a href="http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/">Thunderbird Nightly</a>.</li> <li>Start Thunderbird.</li> <li>Select <kbd>Tools</kbd> → <kbd>Allow Remote Debugging</kbd>.</li> </ul> <figure> <p><img src="/assets/images/thunderbird-allow-remote-debugging.png" alt="Thunderbird - Allow remote debugging" /></p> <figcaption>Thunderbird allow remote debugging</figcaption> </figure> <ul> <li>Download the same version of <a href="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/">Firefox Nightly</a> as you did Thunderbird.</li> <li>Start Firefox.</li> <li>Press <kbd>Ctrl</kbd> → <kbd>Shift</kbd> → <kbd>K</kbd> to open the toolbox.</li> <li>Click <kbd>&#9881;</kbd> to open the developer tools options panel.</li> <li> <p>Check <kbd>Enable Chrome Debugging</kbd> and <kbd>Enable Remote Debugging</kbd> then click <kbd>Restart Now</kbd> if it appears.</p> <figure> <p><img src="/assets/images/enable-chrome-debugging-and-enable-remote-debugging-options.png" alt="Enable Chrome Debugging and Enable Remote Debugging options" /></p> <figcaption>Enable chrome debugging and enable remote debugging options</figcaption> </figure> </li> <li>Select <kbd>Web Developer</kbd> → <kbd>Connect…</kbd></li> <li>Choose localhost port 6000 and click “Connect.”</li> </ul> <p>You will be presented with a set of available remote tabs and processes. For me they appeared as follows:</p> <figure> <p><img src="/assets/images/available-remote-tabs-and-processes.png" alt="Available remote tabs and processes" /></p> <figcaption>Available remote tabs and processes</figcaption> </figure> <ul> <li>“Your Kobo Order Receipt” - the area in which the selected email is displayed. In this view the following did / did not work: <ul> <li>The console tab may work but was empty when I tested.</li> <li>The inspector, debugger, style editor and profiler work just fine.</li> <li>The network tab was empty.</li> </ul> </li> <li>“Message Summary” is the area in which email summaries are displayed when multiple messages are selected. <ul> <li>The console tab may work but was empty when I tested.</li> <li>The inspector, debugger, style editor and profiler work just fine.</li> <li>The network tab was empty.</li> </ul> </li> <li>“Main Process” - the main Thunderbird process. In this view the following did / did not work: <ul> <li>The console, debugger and profiler work fine.</li> <li>The inspector and style editor and network tabs were empty.</li> </ul> </li> </ul> <p>Although not all of the tools work perfectly in all contexts at least you can now use the DevTools for extension development and to fix bugs.</p> <p>Philipp has the same code packaged as an extension that could be used to enable the debug server from other XUL apps <a href="https://addons.mozilla.org/en-US/thunderbird/addon/remote-developer-tools-server/">here</a>.</p> <p>Something has been needed since Venkman development halted so now we have something there is no excuse, whether it is Thunderbird itself or an extension… go, get hacking!</p> Tue, 08 Oct 2013 10:33:00 +0100 http://flailingmonkey.com/debugging-thunderbird-using-firefox-developer-tools/ http://flailingmonkey.com/debugging-thunderbird-using-firefox-developer-tools/ The History of Firebug Mike Ratcliffe <p>When Joe Hewitt was seven years old his father wanted to become a computer programmer so he bought an IBM PCjr. Joe would often watch his father typing in programs from PC Magazine. He started learning himself and when he got older he just sort of took off on his own.</p> <p>The first programming project that he remembers working on was when he was in fifth grade (ten – eleven years old). He made an animated ASCII art of a rocket ship blasting off. He also wrote a game show in “Cartridge Basic” (an enhanced version of BASIC for the IBM PCjr).</p> <p>As far as education goes, he started as a computer science major before deciding that he didn’t like the formal computer science stuff and switched to graphic design. He then realized that he was not an artist and moved back to computer science. Whilst he was still at college he received a ton of job offers in the field, dropped out of college and has been professional ever since.</p> <p>Around 1999 he was working as a web developer and began to gain an interest in Mozilla. He followed the project mostly from the standpoint of a web developer wanting to see what this new browser was going to turn into. Eventually he learned about XUL, became fascinated with it and started looking for ways to get involved.</p> <p>Luckily, before he even really started contributing, a job opened up with Netscape and he was hired to work on the Netscape user interface (CSS, JavaScript and XUL).</p> <p>After a few years at Netscape Joe, Dave Hyatt, Chanial and Blake Ross started working on a streamlined version of the Mozilla browser that was called Phoenix, Firebird and eventually Firefox.</p> <p>Because of the structure of Mozilla the developers were free to work on the things that were of interest to them. During his time there he created the DOM Inspector and worked for a few years on the browser side of things.</p> <p>After his time with Mozilla he started work as a web developer again and soon found that the state of the art tools that he had used as a web developer in the late 90s had not changed. The standard way to debug a web application was using alert dialogs but this was very inefficient and limiting. Joe thought that maybe allowing people to log information to a simple panel could be useful. This is when the birth of modern web developer tools truly began.</p> <p>Joe says that features that you see in Firebug today were inspired by the Venkman JavaScript Debugger, View Rendered Source Chart, Console2, Aardvark, and MochiKit JavaScript Interpreter. Let’s take a look at these tools and see how the functionality were absorbed into Firebug.</p> <h2 id="venkman-javascript-debugger">Venkman JavaScript Debugger</h2> <p>A lot of developers are surprised to hear that Firebug is not the first JavaScript debugger for Firefox. Venkman JavaScript Debugger was, until mid 2011, commonly used to debug JavaScript in web pages and extensions.</p> <figure> <p><img src="/assets/images/venkman-debugger.gif" alt="Venkman Debugger" /></p> <figcaption>Venkman debugger</figcaption> </figure> <p>Venkman contained a very powerful feature set that made for an excellent debugger. These included:</p> <ul> <li>Stop</li> <li>Continue</li> <li>Step over</li> <li>Step into</li> <li>Step out</li> <li>JavaScript profiler</li> <li>Save profile data</li> <li>Pretty print</li> <li>Save and restore breakpoints and watches</li> <li>Break on errors</li> <li>Ignore errors</li> <li>Break on exceptions</li> <li>Ignore exceptions</li> <li>Function name guessing</li> <li>Locals</li> <li>Watches</li> <li>Breakpoints</li> <li>Call stack</li> </ul> <p>As this is just a quick tour of the extensions that have inspired Web Developer Tools we will not go into any more detail about Venkman here. If you would like to learn more about it then take a look at the <a href="https://developer.mozilla.org/docs/Venkman">Venkman page</a> on MDC.</p> <p>Venkman was the inspiration for the JavaScript Debugger used in Firebug.</p> <h2 id="view-source-chart">View Source Chart</h2> <p>This extension shows the browsers interpretation of a web page. It shows the source in a hierarchical view, grouping elements at the same level and graphically displaying HTML tag boundaries.</p> <figure> <p><img src="/assets/images/view-source-chart.png" alt="View Source Chart" /></p> <figcaption>View Source Chart</figcaption> </figure> <p>This was the inspiration for Firebug’s HTML panel … the comparison is clear when both tools are compared, especially in early Firebug versions. The formatting is fairly similar except that Firebug has a number of improvements such as expandable and collapsible HTML nodes, improved indentation and live editing. In its time, view rendered source was a very useful way to view the structure of an HTML page and is still preferred by some developers.</p> <figure> <p><img src="/assets/images/firebug-html-tree.png" alt="Firebug HTML Tree" /></p> <figcaption>Firebug HTML Tree</figcaption> </figure> <h2 id="console2">Console<sup>2</sup></h2> <figure> <p><img src="/assets/images/console2.png" alt="Console2" /></p> <figcaption>Console2</figcaption> </figure> <p>Console<sup>2</sup>, pronounced Console Squared or Console Two, replaced the JavaScript Console with what was a revolutionary Error Console.</p> <p>This extension made it much simpler to find the cause of JavaScript errors by showing the line number and, eventually, allowing you to link to this line in the source. This method is all that is required to find syntax errors but is not so useful when it comes to finding logic errors. Still, this extension saved developers many, many hours when debugging JavaScript.</p> <p>This extension was the inspiration for the error reporting in Firebug’s Console Panel.</p> <h2 id="aardvark">Aardvark</h2> <p>Aardvark was another excellent extension and also the inspiration for Firebug’s Inspector. By moving the mouse over elements on the screen you could view simple DOM properties like tag name, class and id.</p> <figure> <p><img src="/assets/images/aardvark.png" alt="Aardvark" /></p> <figcaption>Aardvark</figcaption> </figure> <p>Sometimes the simplest ideas are the best and the concept of the inspector (highlighter) quickly became one of the core features of Firebug as it made it possible for people to select HTML elements as targets for each of Firebug’s panels.</p> <h2 id="mochikit-javascript-interpreter">MochiKit JavaScript Interpreter</h2> <p><a href="http://mochi.github.com/mochikit/">MochiKit</a> is a lightweight JavaScript library written and maintained by Bob Ippolito. It is Inspired by the Python networking framework and Twisted. It uses the concept of deferred execution to allow asynchronous behavior. This has made it very useful in the development of AJAX applications.</p> <p>Of particular note is its ability to load and manipulate JSON-encoded data sets, and MochiKit.DOM, a set of functions to easily create dynamic page components.</p> <p>MochiKit forms the foundation of the client-side functionality of the TurboGears Python web-application stack. Perhaps as a result of the author’s involvement in the Python community, MochiKit exhibits many idioms familiar to Python programmers, and is commonly used in Python-based web applications.</p> <p>The JavaScript Interpreter allows you to run JavaScript commands a line at a time and view the results in a div. Using this command line it is possible to quickly view the values of DOM properties, change the value of variables or call a pages functions. This was Joe Hewitt’s inspiration for Firebug’s JavaScript Interpreter (part of the JavaScript Console).</p> <figure> <p><img src="/assets/images/mochikit-javascript-interpreter.png" alt="MochiKit JavaScript Interpreter" /></p> <figcaption>Mochikit JavaScript interpreter</figcaption> </figure> <h2 id="firebug-through-the-ages">Firebug Through the Ages</h2> <p>Although all of the applications we have discussed so far in this chapter were inspiration for some of Firebug’s features, this does not mean that they were simply merged together and Firebug was complete. Firebug, like most applications was very simple and has gradually evolved into the fantastic tool that it is now.</p> <h2 id="version-02">Version 0.2</h2> <iframe src="http://www.youtube.com/embed/JVCioNT-SYE?fs=1" allowfullscreen=""></iframe> <p>The first publicly available release of Firebug was version 0.2. Although this was the first released version of Firebug it contained some very useful features even though there was only a single panel. In this version of Firebug there is only one panel, which is roughly equivalent to the console panel. AJAX logging was present even in this version.</p> <p>The printfire function was the standard way to output messages. Inspected elements were logged to the panel with four tabs on the right hand side. Clicking these tabs expanded the panel’s representation of the element, which consisted of:</p> <ul> <li>The XML tab shows the element in HTML view</li> <li>The CSS tab shows the elements CSS styles</li> <li>The Box tab shows the selected elements classic box model property values</li> <li>The JS tab shows the JavaScript properties of the selected element</li> </ul> <p>Whenever there were JavaScript errors the status bar indicator would display the number of errors. Clicking on this status bar indicator would open Firebug.</p> <figure> <p><img src="/assets/images/firebug-02-status-bar-indicator.png" alt="Firebug 0.2 status bar indicator" /></p> <figcaption>Firebug 0.2 status bar indicator</figcaption> </figure> <h2 id="version-03">Version 0.3</h2> <p>Version 0.3 introduced major changes to the way that Firebug worked, particularly the way that the inspector worked. Firebug now had two main tabs, a console tab to display messages and an inspector tab that allowed you to view the element. There was also a new search box that allowed you to search for tabs in the HTML Tree. The Inspector tab had five other tabs:.</p> <ul> <li>Source: The HTML tree. This panel has been largely unchanged from this version onwards.</li> <li>Style: A few computed styles.</li> <li>Layout: Basic layout information.</li> <li>Events: Any events raised triggered on the currently selected element.</li> <li>DOM: JavaScript object view of the currently selected node.</li> </ul> <iframe src="http://www.youtube.com/embed/HBHkanS1euM?fs=1" allowfullscreen=""></iframe> <h2 id="version-04">Version 0.4</h2> <p>Version 0.4 was the first version to feature the Debugger tab (figure 2.7.7.1).</p> <ul> <li>Stack traces were now displayed along with error messages.</li> <li>console.log(‘message’), console.warn(‘message’), console.info(‘message’) etc. replaced printfire and were made available to all web pages.</li> <li>In the DOM tab arrays were now expandable and functions could be expanded to show their source.</li> </ul> <p>Obviously, the inclusion of the JavaScript debugger was a major step forwards in Firebug’s evolutionary trail. The whole idea of debugging JavaScript in the browser itself was something we had been waiting a long time for. Great job Joe!</p> <iframe src="http://www.youtube.com/embed/LvdcAm1-4zU?fs=1" allowfullscreen=""></iframe> <h2 id="version-10">Version 1.0</h2> <p>After version 0.4.1 Joe realized that people were depending on Firebug for their jobs and decided to have one person work on it full time. Version 1.0 represented a complete redesign of the Firebug user interface from the ground up. A bunch of new features were also released with this version, Joe lists them as:</p> <ul> <li>CSS editing</li> <li>Network load timing</li> <li>Box model visualization</li> <li>JavaScript profiling</li> <li>Command line autocomplete</li> <li>HTML change highlighting</li> <li>Debugger watchlists</li> <li>DOM editing</li> <li>Separate window support</li> <li>Per-site blacklists</li> </ul> <iframe src="http://www.youtube.com/embed/Ks_Pp2jghVY?fs=1" allowfullscreen=""></iframe> <p>At this point Joe was spending a lot of time on Firebug and was considering making it a paid product ($15 - $25) so that he could employ a couple of full-time engineers. Needless to say, this was a final option and never happened.</p> <p>Although there has been lots of work done on Firebug since version 1.0 most of it has been bugfixes and minor features. Some major changes have crept in though, including:</p> <ul> <li>Console tab: Break on all errors.</li> <li>HTML tab: Break on HTML mutation.</li> <li>DOM tab: Break on property change.</li> <li>Net tab: Break on XHR.</li> <li>Persist button on all tabs.</li> <li>Switch color types.</li> <li>Restartless install</li> <li>Tabs on the top</li> <li>Navigation history</li> <li>Break on next</li> <li>Cookie tab</li> <li>Command line popup</li> <li>Command line history</li> <li>New side panels (e.g. selectors)</li> <li>List of tabs in the toolbar</li> <li>Tab options</li> </ul> <p>Joe Hewitt talks about these things in his blog post <a href="http://joehewitt.com/2006/03/15/firebug-a-love-story">Firebug: A Love Story</a>.</p> Wed, 25 Sep 2013 10:22:00 +0100 http://flailingmonkey.com/the-history-of-firebug/ http://flailingmonkey.com/the-history-of-firebug/ Why it is awesome to work for Mozilla? Mike Ratcliffe <p><img src="/assets/images/mike-caricature.jpg" alt="Mike Caricature" /></p> <p>Since my post about <a href="/what-is-it-like-to-work-as-an-engineer-at-mozilla/">what it is like to be a Firefox hacker</a> lots of people have asked about why I chose to work for Mozilla. I had opportunities to work for Google or Facebook but I have always had a great respect for Mozilla.</p> <p>Of course, Google and Facebook are also creative and constantly working on awesome stuff but Mozilla is more than just a job… their mission is something that I feel that I really need to fight for.</p> <p>Back in 1997 the Netscape browser was pushing more and more traffic towards AOL’s search and other services. It is never any good for a single company to have too much power and control over the internet and if things continued as they were, then it is likely that one company would have “owned” the internet. Could you imagine everybody having to pay a single company for internet access, storing contacts, searches and each page view?</p> <p>Eventually, Mozilla broke away from AOL Netscape because they believed that the internet should be free and open for everybody. <a href="http://www.mozilla.org/mission/">Mozilla’s mission</a> is not to simply write and maintain Firefox, it is to keep the power of the Web in people’s hands. Firefox is simply a means to an end in accomplishing this.</p> <p>Mozilla does a fantastic job of promoting openness, innovation and opportunity on the Internet. They do this by teaching kids and adults to create web pages, inventing awesome stuff, contributing to web standards and generally trying to make the web a better place. They really are building the future of the internet.</p> <p>Mozilla also does a great job of keeping peoples data private. Countless people have asked how this can be when sync data is stored on Mozilla’s servers. We have made huge efforts to keep this data private. All of the data is stored in a gigantic unreadable blob. Each computer has a key that it uses to access the data and without that key the data is completely useless.</p> <p>So why is it awesome to work for Mozilla? Well, I am working with some of the most talented, creative developers in the world. The internet is still at risk of becoming “owned” by large organisations and I don’t like this idea and I refuse to go down without a fight. I refuse to hand all of my data over to those that ask nicely enough by offering “free” services in exchange for my privacy. Everything I do at work helps keep the internet free and open to everybody and this helps me feel that I am fighting to save a free and open internet.</p> <p>Amazingly, the right to privacy and a desire to make the internet a better place is only one reason that it is awesome to work for Mozilla. Another thing is that my mindset is set squarely within the open source culture. Open source simply produces better software… everyone collaborates, a product evolves at an astonishing rate and the best technology wins. Given enough eyeballs, all bugs are shallow.</p> <p>I also think that there is a place for paid software. If somebody is working hard to provide a quality service then I am more than happy to pay, even if it is open source. There is something very appealing about being able to look at a program’s source to make sure a program is doing what it claims to be. Submitting fixes and adding new features to these programs is also something that feels right… it just feels like this is the way things are supposed to work.</p> <p>The icing on the cake has to be that everyone that works for Mozilla feels the same way about open source software. There is a unity of purpose at the company that I do not believe exists anywhere else. We don’t create awesome stuff just because we have been asked to do it. We create it because we have an awesome idea and have che chance to bring it to life. We code for the joy of coding and no job gets better than that.</p> <p>Amazingly <a href="http://www.whatcanidoformozilla.org/">everybody can do something for Mozilla</a> and join the fight keep the internet open… you don’t even need to know any programming languages.</p> Sun, 15 Sep 2013 10:10:00 +0100 http://flailingmonkey.com/why-it-is-awesome-to-work-for-mozilla/ http://flailingmonkey.com/why-it-is-awesome-to-work-for-mozilla/ Bugzilla Tips, Hacks and Etiquette Mike Ratcliffe <p>Even after a few years working for Mozilla I still occasionally come across a Bugzilla tip that I don’t know about. This post is a attempt to gather Bugzilla tips from around the Internet whilst adding a few of my own to the list.</p> <p>To start, if you’ve never used Bugzilla before (or even if you have), you should watch a video <a href="https://twitter.com/johnath">johnath</a> made called “Bugzilla for Humans.” It gives a really nice overview of key Bugzilla features, as well as tips for more advanced users:</p> <iframe src="http://www.youtube.com/embed/piT1pflOTDs?fs=1" allowfullscreen=""></iframe> <h2 id="quicksearch-ill-race-ya">Quicksearch, I’ll Race Ya!</h2> <p>The most useful tip I know of is to set up a keyword search for the quicksearch box. Open <a href="https://bugzilla.mozilla.org/">https://bugzilla.mozilla.org/</a> and right-click the quicksearch box near the top of the page. Select “Add a keyword for this search…” and in the keyword field type “bug.” You can now search bugzilla using queries like this directly from your Firefox Awesome Bar:</p> <p><code class="highlight language-javascript"><span class="nx">bug</span> <span class="mi">50000</span></code> opens bug 50000.</p> <p>Or even: <code class="highlight language-javascript"><span class="nx">bug</span> <span class="nx">WONTFIX</span> <span class="nx">Inspector</span></code> open bugs containing “Inspector” and marked as WONTFIX.</p> <p>Two lists of quicksearch fields are available. A simplified list can be found <a href="http://www.squarefree.com/bugzilla/quicksearch-help.html">here</a> and a more complete list can be found <a href="https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html">here</a>.</p> <p><a href="http://twitter.com/beltzner">Mike Beltzner</a> created an excellent video explaining some of the ways this feature can be used:</p> <iframe src="http://www.youtube.com/embed/14W-XguG--U?fs=1" allowfullscreen=""></iframe> <h2 id="watching-a-component-or-a-developer">Watching a Component or a Developer</h2> <p>If you find yourself interested in a certain area of the Mozilla project e.g. Firefox::Developer Tools, you can opt to <a href="https://bugzilla.mozilla.org/userprefs.cgi?tab=component_watch">watch its Bugzilla component</a>. You can also <a href="https://bugzilla.mozilla.org/userprefs.cgi?tab=email">watch certain users</a>, which is a really neat feature if you’re a new contributor looking to follow the activity of a more experienced developer.</p> <h2 id="autolinkification">Autolinkification</h2> <p>Bugzilla comments are plain text - so typing &lt;U&gt; will produce less-than, U, greater-than rather than underlined text. However, Bugzilla will automatically make hyperlinks out of certain sorts of text in comments. For example, the text “http://www.bugzilla.org” will be turned into a link: http://www.bugzilla.org. Other strings which get linkified in the obvious manner are:</p> <ol> <li>bug 12345</li> <li>comment 7</li> <li>bug 23456, comment 53</li> <li>attachment 4321</li> <li>mailto:george@example.com</li> <li>george@example.com</li> <li>ftp://ftp.mozilla.org</li> <li>Most other sorts of URL</li> </ol> <p>A corollary here is that if you type a bug number in a comment, you should put the word “bug” before it, so it gets autolinkified for the convenience of others.</p> <h2 id="use-needinfo">Use Needinfo</h2> <p>There is a flag for all bugs in Bugzilla called the needinfo flag which should be used if you need information from someone. So if you are asking someone a question, then needinfo them!</p> <h2 id="is-your-bugzilla-butt-ugly">Is your Bugzilla Butt-Ugly?</h2> <p>You are probably still using the default theme. This can be easily remedied:</p> <ul> <li>Click <kbd>Preferencies</kbd> → <kbd>General Preferences</kbd>.</li> <li>Under <kbd>Bugzilla's general appearance (skin)</kbd> select <kbd>Mozilla</kbd></li> </ul> <p>Now your Bugzilla looks <a href="http://www.youtube.com/watch?v=R2iOCHhJVhE">sexy and you know it</a>!</p> <h2 id="does-your-bugmail-make-your-eyes-bleed">Does your Bugmail make your Eyes Bleed?</h2> <p>Well, you probably have Bugzilla set up to use the default email format (text):</p> <figure> <p><img src="/assets/images/bugzilla-plain-text-email.jpg" alt="Bugzilla Plain Text Email" /></p> <figcaption>Bugzilla plain text email</figcaption> </figure> <p>Switching this to HTML format makes them 1000 times easier to read, especially on a phone. Don’t look for it under the “Email Preferences” tab though, that would be way too easy.</p> <ul> <li>Click <kbd>Preferencies</kbd> → <kbd>General Preferences</kbd>.</li> <li>Under <kbd>Preferred email format</kbd> select <kbd>HTML</kbd>.</li> </ul> <figure> <p><img src="/assets/images/bugzilla-html-email.jpg" alt="Bugzilla HTML Email" /></p> <figcaption>Bugzilla HTML email</figcaption> </figure> <h2 id="bugzilla-extensions-to-rock-your-world">Bugzilla Extensions to Rock your World</h2> <p>There are a few great extensions available that make Bugzilla a more pleasurable experience. Here are some of my favorites:</p> <ol> <li><a href="https://addons.mozilla.org/firefox/addon/bugzillajs/">BugzillaJS</a>: Created by <a href="https://twitter.com/gkoberger">Gregory Koberger</a> and <a href="https://twitter.com/rik24d">Anthony Ricaud</a>. This extension adds keyboard shortcuts, quick file box, inline image / github previews, relative timestamps, Gravatars and more. More information about features is available on the extensions AMO page. This extension contains almost all of Bugzilla Tweak’s features and more. You can contribute to the addon <a href="https://github.com/gkoberger/BugzillaJS">here</a></li> <li><a href="https://addons.mozilla.org/firefox/addon/bugzilla-tweaks/">Bugzilla Tweaks</a>: Created by <a href="https://twitter.com/ehsanakhgari">Ehsan Akhgari</a>, Dave Lawrence and <a href="https://twitter.com/globau">Byron Jones</a>. This extension adds a bunch of features that do not appear to be documented anywhere. These are features such as adding “Copy Check-in Comment” to the context menu. If anybody knows how to contribute to this extension then please let me know.</li> <li><a href="https://addons.mozilla.org/firefox/addon/github-tweaks-for-bugzilla/">Github tweaks for Bugzilla</a>: Created by <a href="https://twitter.com/dietrich">Dietrich Ayala</a>. This extension allows for better Github and Bugzilla integration. On a pull-request page, it adds a new button for submitting the pull-request as an attachment to a bug in bugzilla.mozilla.org. The bug id can be specified by starting your pull request title with “bug ######”. Your Bugzilla credentials are pulled out of Firefox’s password manager. You can contribute to this extension <a href="https://github.com/autonome/Github-Bugzilla-Tweaks">here</a>.</li> <li><a href="http://harthur.github.io/bugzilla-todos/">Bugzilla TODOs</a>: This is not an extension but a really useful live dashboard created by <a href="https://twitter.com/harthvader">Heather Arthur</a>. It contains five tabs (To Review, To Check In, To Nag, To Fix and To Respond) and, because it is live, it is updated automagically. I keep this open as a pinned tab so that I can instantly (via the favicon) see if somebody wants something from me. You can contribute to this handy little web app <a href="https://github.com/harthur/bugzilla-todos">here</a>.</li> </ol> <h2 id="attachments">Attachments</h2> <p>Use attachments, rather than comments, for large chunks of ASCII data, such as trace, debugging output files, or log files. That way, it doesn’t bloat the bug for everyone who wants to read it, and cause people to receive fat, useless mails.</p> <p>Trim screenshots. There’s no need to show the whole screen if you are pointing out a single-pixel problem.</p> <p>Don’t attach simple test cases (e.g. one HTML file, one CSS file and an image) as a ZIP file. Instead, upload them in reverse order and edit the referring file so that they point to the attached files. This way, the test case works immediately out of the bug.</p> <p>Bugzilla stores and uses a Content-Type for each attachment (e.g. text/html). To download an attachment as a different Content-Type (e.g. application/xhtml+xml), you can override this using a ‘content-type’ parameter on the URL e.g. <code class="highlight language-javascript"><span class="o">&amp;</span><span class="nx">content</span><span class="o">-</span><span class="nx">type</span><span class="o">=</span><span class="nx">text</span><span class="o">/</span><span class="nx">plain</span><span class="p">.</span></code></p> <h2 id="etiquette">Etiquette</h2> <p>If you are changing the fields on a bug, only comment if either you have something pertinent to say, or Bugzilla requires it. Otherwise, you may spam people unnecessarily with bug mail. To take an example: a user can set up their account to filter out messages where someone just adds themselves to the CC field of a bug (which happens a lot.) If you come along, add yourself to the CC field, and add a comment saying “Adding self to CC”, then that person gets a pointless piece of mail they would otherwise have avoided.</p> <p>Don’t use sigs in comments. Signing your name (“Bill”) is acceptable, if you do it out of habit, but full mail/news-style four line ASCII art creations are not.</p> <p>You should also read <a href="https://bugzilla.mozilla.org/page.cgi?id=etiquette.html">Bugzilla’s etiquette guidelines</a> to learn what behaviors are acceptable for Bugzilla users. In general, just be a nice person and everyone will be happy!</p> <h2 id="credits">Credits</h2> <p>Some of this information was collected from other websites so it is only fair that I give credit where credit is due:</p> <ul> <li><a href="http://blog.margaretleibovic.com/post/36893756730/bugzilla-101">Margaret Leibovic’s Bugzilla 101</a></li> <li><a href="http://blog.johnath.com/2010/02/04/bugzilla-for-humans/">Johnath’s Bugzilla for Humans</a></li> <li><a href="http://bugzillatips.wordpress.com/">Bugzilla Tips Wordpress Blog</a></li> <li><a href="http://www.squarefree.com/bugzilla/quicksearch-help.html">Jesse Ruderman’s Quicksearch Tips</a></li> <li><a href="https://bugzilla.mozilla.org/page.cgi?id=quicksearch.html">Bugzilla Quicksearch</a></li> <li><a href="http://www.bugzilla.org/docs/2.18/html/hintsandtips.html">The Bugzilla Guide - Hints and Tips</a></li> </ul> <p>If the comments section below has not loaded below then you probably need to update your browser.</p> Thu, 27 Jun 2013 10:02:00 +0100 http://flailingmonkey.com/bugzilla-tips-hacks-and-etiquette/ http://flailingmonkey.com/bugzilla-tips-hacks-and-etiquette/ Disable JavaScript option now available in Firefox Developer Tools Mike Ratcliffe <p>At the beginning of April <a href="http://limi.net/">Alex Limi</a> decided that we should <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=851702">remove the Enable JavaScript checkbox</a> from Firefox’s preferences dialog. His reasoning was that clearing that checkbox breaks a huge amount of websites and therefore breaks the web.</p> <p>Alex was completely correct and I wholeheartedly agree with the removal of this option but if you look at the bug’s comments you will see that a bunch of people are not so happy with this decision. Well, we have news for you. We have just landed a change that <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=864249">adds a tab specific version of this option</a> to the Firefox Developer Tools preferences panel.</p> <figure> <p><img src="/assets/images/disable-javascript-option-in-firefox-devtools.jpg" alt="Disable JavaScript Option in Firefox DevTools" /></p> <figcaption>Disable javascript option in firefox devtools</figcaption> </figure> <p>If you check the box then the page will refresh and, from that point on, everything that you open in that tab will have JavaScript disabled. Making this setting per tab makes sense because the developer toolbox has one instance per tab. Closing either the toolbox or the tab enables JavaScript again.</p> Thu, 30 May 2013 09:59:00 +0100 http://flailingmonkey.com/disable-javascript-option-now-available-in-firefox-developer-tools/ http://flailingmonkey.com/disable-javascript-option-now-available-in-firefox-developer-tools/ The Application Cache is no longer a Douchebag Mike Ratcliffe <p>If you have ever used the application cache you will probably be familiar with how easy it is to break offline functionality and how difficult it is to work out why things are broken. With a large offline web app finding the source of problems can be an almost insurmountable task.</p> <p>I guess that what I am saying is that <a href="http://alistapart.com/article/application-cache-is-a-douchebag">the application cache is a douchebag</a>. I completely understand that if there is an error in your application’s manifest you probably don’t want your users to be confronted with a bunch of error messages but something needs to be done.</p> <p>Because displaying a bunch of errors about a broken cache manifest is probably undesirable we have decided to create a developer tool that lets you know if you have a problem with your manifest. The main problem we have is that the Toolbox is already becoming crowded and we are just getting started.</p> <figure> <p><img src="/assets/images/firefox-developer-tools-toolbox.jpg" alt="Firefox Developer Tools Toolbox" /></p> <figcaption>Firefox developer tools toolbox</figcaption> </figure> <p>In order to avoid cluttering the Toolbox we have used the Firefox Command Line. The command line is a place where we can add as many tools as we like without cluttering the interface. To open the Developer Toolbar press <kbd>shift</kbd> + <kbd>f2</kbd> or select <kbd>Developer Toolbar</kbd> from the web developer menu.</p> <p>The appcache commands are available in <a href="http://nightly.mozilla.org/">Firefox Nightly</a></p> <p>To list available appcache commands type “help appcache” in the developer toolbar.</p> <figure> <p><img src="/assets/images/output-from-the-help-appcache-command.jpg" alt="Output from the help appcache command" /></p> <figcaption>Output from the help appcache command</figcaption> </figure> <p>The currently available appcache commands are:</p> <p><code class="highlight language-javascript"><span class="nx">appcache</span> <span class="nx">clear</span></code> clears all appcache entries.</p> <p><code class="highlight language-javascript"><span class="nx">appcache</span> <span class="nx">list</span> <span class="p">[</span><span class="nx">searchterm</span><span class="p">]</span></code> lists all appcache entries containing an optional search term</p> <figure> <p><img src="/assets/images/output-from-the-appcache-list-command.jpg" alt="Output from the appcache list command" /></p> <figcaption>Output from the appcache list command</figcaption> </figure> <p><code class="highlight language-javascript"><span class="nx">appcache</span> <span class="nx">viewentry</span></code> shows the about:cache-entry page for the current cache entry.</p> <p>Note: appcache viewentry will not give you any information until <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=861866">bug 861866</a> is fixed.</p> <p><code class="highlight language-javascript"><span class="nx">appcache</span> <span class="nx">validate</span> <span class="p">[</span><span class="nx">URI</span><span class="p">]</span></code> is by far the most useful of the appcache commands. If a URI is specified then the cache manifest for that page will be validated. As many of you know, there are a lot of things that could invalidate a manifest but this command will uncover most of them.</p> <figure> <p><img src="/assets/images/output-of-the-appcache-validate-command.jpg" alt="Output of the appcache validate command" /></p> <figcaption>Output of the appcache validate command</figcaption> </figure> <p>If you find any situations where an application cache is invalid and our tool does not pick up the problem then please <a href="https://bugzilla.mozilla.org/enter_bug.cgi?alias=&amp;assigned_to=nobody%40mozilla.org&amp;attach_text=&amp;blocked=&amp;bug_file_loc=http%3A%2F%2F&amp;bug_severity=normal&amp;bug_status=NEW&amp;cf_blocking_b2g=---&amp;cf_crash_signature=&amp;cf_status_b2g18=---&amp;cf_status_b2g18_1_0_0=---&amp;cf_status_b2g18_1_0_1=---&amp;cf_status_firefox20=---&amp;cf_status_firefox21=---&amp;cf_status_firefox22=---&amp;cf_status_firefox23=---&amp;cf_status_firefox_esr17=---&amp;cf_tracking_b2g18=---&amp;cf_tracking_firefox20=---&amp;cf_tracking_firefox21=---&amp;cf_tracking_firefox22=---&amp;cf_tracking_firefox23=---&amp;cf_tracking_firefox_esr17=---&amp;cf_tracking_firefox_relnote=---&amp;cf_tracking_relnote_b2g=---&amp;comment=&amp;component=Developer%20Tools%3A%20Graphic%20Commandline%20and%20Toolbar&amp;contenttypeentry=&amp;contenttypemethod=autodetect&amp;contenttypeselection=text%2Fplain&amp;data=&amp;defined_groups=1&amp;dependson=&amp;description=&amp;flag_type-203=X&amp;flag_type-37=X&amp;flag_type-41=X&amp;flag_type-5=X&amp;flag_type-607=X&amp;flag_type-720=X&amp;flag_type-721=X&amp;flag_type-737=X&amp;flag_type-748=X&amp;flag_type-781=X&amp;flag_type-787=X&amp;flag_type-791=X&amp;flag_type-799=X&amp;flag_type-800=X&amp;flag_type-802=X&amp;flag_type-809=X&amp;form_name=enter_bug&amp;keywords=&amp;maketemplate=Remember%20values%20as%20bookmarkable%20template&amp;op_sys=All&amp;priority=--&amp;product=Firefox&amp;qa_contact=&amp;rep_platform=All&amp;requestee_type-203=&amp;requestee_type-41=&amp;requestee_type-5=&amp;requestee_type-607=&amp;requestee_type-748=&amp;requestee_type-781=&amp;requestee_type-787=&amp;requestee_type-791=&amp;requestee_type-800=&amp;short_desc=&amp;status_whiteboard=&amp;target_milestone=---&amp;version=unspecified">let us know</a>. In fact, if you have an idea for a new appcache command then please <a href="https://bugzilla.mozilla.org/enter_bug.cgi?alias=&amp;assigned_to=nobody%40mozilla.org&amp;attach_text=&amp;blocked=&amp;bug_file_loc=http%3A%2F%2F&amp;bug_severity=enhancement&amp;bug_status=NEW&amp;cf_blocking_b2g=---&amp;cf_crash_signature=&amp;cf_status_b2g18=---&amp;cf_status_b2g18_1_0_0=---&amp;cf_status_b2g18_1_0_1=---&amp;cf_status_firefox20=---&amp;cf_status_firefox21=---&amp;cf_status_firefox22=---&amp;cf_status_firefox23=---&amp;cf_status_firefox_esr17=---&amp;cf_tracking_b2g18=---&amp;cf_tracking_firefox20=---&amp;cf_tracking_firefox21=---&amp;cf_tracking_firefox22=---&amp;cf_tracking_firefox23=---&amp;cf_tracking_firefox_esr17=---&amp;cf_tracking_firefox_relnote=---&amp;cf_tracking_relnote_b2g=---&amp;comment=&amp;component=Developer%20Tools%3A%20Graphic%20Commandline%20and%20Toolbar&amp;contenttypeentry=&amp;contenttypemethod=autodetect&amp;contenttypeselection=text%2Fplain&amp;data=&amp;defined_groups=1&amp;dependson=&amp;description=&amp;flag_type-203=X&amp;flag_type-37=X&amp;flag_type-41=X&amp;flag_type-5=X&amp;flag_type-607=X&amp;flag_type-720=X&amp;flag_type-721=X&amp;flag_type-737=X&amp;flag_type-748=X&amp;flag_type-781=X&amp;flag_type-787=X&amp;flag_type-791=X&amp;flag_type-799=X&amp;flag_type-800=X&amp;flag_type-802=X&amp;flag_type-809=X&amp;form_name=enter_bug&amp;keywords=&amp;maketemplate=Remember%20values%20as%20bookmarkable%20template&amp;op_sys=All&amp;priority=--&amp;product=Firefox&amp;qa_contact=&amp;rep_platform=All&amp;requestee_type-203=&amp;requestee_type-41=&amp;requestee_type-5=&amp;requestee_type-607=&amp;requestee_type-748=&amp;requestee_type-781=&amp;requestee_type-787=&amp;requestee_type-791=&amp;requestee_type-800=&amp;short_desc=&amp;status_whiteboard=&amp;target_milestone=---&amp;version=unspecified">tell us</a>.</p> Sun, 05 May 2013 09:52:00 +0100 http://flailingmonkey.com/application-cache-not-a-douchebag/ http://flailingmonkey.com/application-cache-not-a-douchebag/ Moving your Contacts and Calendar Away from Google Mike Ratcliffe <p><img src="/assets/images/freedom.jpg" alt="Freedom" /></p> <p>For the past few years I have used Google Contacts to store my contacts and Google Calendar to store my calendars. These services really are excellent, particularly because they allow me to keep my data synchronized between my Android phone and numerous instances of Thunderbird on PCs and Laptops. Unfortunately, I have had issues with my contacts becoming duplicated and even disappearing altogether. I figure that out of roughly 500 contacts I have lost around 200. This really got me thinking, I mean, should I really rely on somebody else to look after my data?</p> <p>I don’t know exactly why Google run these services but that is part of the problem. They are an advertising company, after all, and the more data they can collect about me and the people I know the better they can target me with their adverts. Maybe I am paranoid but now that they are dropping Google Reader I just don’t trust them to the extent that I once did.</p> <p>Don’t get me wrong, I am not a Google hater, I truly believe that their search engine is excellent and that nothing else comes close to providing such accurate search results, at least when I am not logged in. There are a number of services that allow this kind of synchronization but I have chosen to use <a href="https://owncloud.org/" title="ownCloud">ownCloud</a> as it appears to be the most capable open source solution. I have included installation instructions here for the good of the web.</p> <h2 id="installing-owncloud">Installing ownCloud</h2> <h3 id="if-you-own-your-own-server">If you own your own server</h3> <p>Follow the instructions <a href="http://software.opensuse.org/download/package?project=isv:ownCloud:community&amp;package=owncloud" title="Installing ownCloud on your own server">here</a>.</p> <h3 id="shared-hosting">Shared hosting</h3> <ul> <li>In your hosts control panel select the latest version of PHP, for JustHost this is in CPanel under PHP Config.</li> <li>Right-click <a href="https://download.owncloud.com/download/community/setup-owncloud.php" title="ownCloud installer">here</a> and save the file to your computer.</li> <li>Copy the file to http://yourdomain.com/setup-owncloud.php</li> <li>Open http://yourdomain.com/setup-owncloud.php in your browser.</li> <li>Add a username and password but <strong>do not click “Next.”</strong></li> <li>Click <kbd>Advanced</kbd> and select your database options. A MySQL DB will be way faster than a SQLite DB.</li> <li>Accept the default options.</li> </ul> <h2 id="exporting-contacts-from-google-contacts">Exporting Contacts from Google Contacts</h2> <p>At the time of writing, exporting contacts from Google Contacts as a .vcf file only exports one name, phone number and email address per contact. Fortunately, if you export your contacts from an Android phone you will keep all of your contact information including mugshots. To do this:</p> <ul> <li>From a Home or All Apps screen, touch the People icon.</li> <li>Select <kbd>Menu</kbd> → <kbd>Import / Export</kbd>.</li> <li>Select <kbd>Export to storage</kbd>.</li> <li>Touch “OK” to confirm.</li> </ul> <p>A file with the .vcf extension will be saved to the root directory of your phone’s internal storage. This file contains all of your contacts including their photos. Transfer this file to your computer via a USB cable or email.</p> <ul> <li>Go to http://yourdomain.com/owncloud/</li> <li>Click <kbd>Contacts</kbd> → <kbd>Import</kbd>.</li> <li>Select the file .vcf file that you copied to your computer and click OK to import.</li> </ul> <h2 id="importing-calendar-to-owncloud">Importing Calendar to ownCloud</h2> <ul> <li>Go to <a href="http://www.google.com/calendar/" title="Google Calendar">http://www.google.com/calendar/</a></li> <li>Click <kbd>&#9881;</kbd> → <kbd>Settings</kbd> → <kbd>Calendars</kbd> → <kbd>Export Calendar</kbd>.</li> <li>Save the .ics file to your computer.</li> <li>Go to http://yourdomain.com/owncloud/</li> <li>Click Files and drag the .ics file into the files window in order to upload it.</li> <li>Click on the .ics file.</li> <li>In the Import calendar dialog that appears select “Default calendar” and click import.</li> <li>When the calendar has been imported close the dialog and delete the .ics file.</li> </ul> <h2 id="configure-thunderbird">Configure Thunderbird</h2> <h3 id="calendar">Calendar</h3> <ul> <li>Uninstall the Provider for Google Calendar add-on.</li> <li>Install <a href="http://www.sogo.nu/files/downloads/SOGo/Thunderbird/" title="SOGo Connector">SOGo Connector</a>.</li> <li>Install <a href="http://nic-nac-project.org/~kaosmos/morecols-en.html" title="MoreFunctionsForAddressBook">MoreFunctionsForAddressBook</a>.</li> <li>Install the <a href="https://addons.mozilla.org/en-US/thunderbird/addon/lightning/" title="Lightning">Lightning extension</a>.</li> <li>In your owncloud calendar copy your calendar’s CalDAV link.</li> <li>Back in Thunderbird open the calendar.</li> <li>Click <kbd>File</kbd> → <kbd>New Calendar</kbd> → <kbd>On the Network</kbd> → <kbd>Next</kbd> → <kbd>CalDAV</kbd>.</li> <li>Type the following into the location field:</li> <li>http://yourdomain/owncloud/remote.php/caldav/calendars/USERNAME/CALENDARNAME (the default calendar name is defaultcalendar).</li> <li>Check <kbd>Offline Support</kbd>.</li> <li>Click <kbd>Next</kbd></li> <li>Give your calendar a name and an email address. The email address is used for email notifications.</li> </ul> <h3 id="contacts">Contacts</h3> <ul> <li>Uninstall the Google Contacts addon.</li> <li>Install <a href="http://www.sogo.nu/files/downloads/SOGo/Thunderbird/" title="SOGo Connector">SOGo Connector</a>.</li> <li>In Thunderbird open the Address Book.</li> <li><kbd>File</kbd> → <kbd>New</kbd> → <kbd>Remote Address Book</kbd>.</li> <li>Give your address book a name.</li> <li>Type the URL into the appropriate field. The URL is found in your OwnCloud Contacts area, that little Gear symbol in the bottom left of the Contacts View (same symbol as found in the top right in the Calendar view). Click on the little world (CardDAV Link) symbol and it will display the URL you need for your installation to work.</li> <li>Click <kbd>OK</kbd>.</li> <li>Right-click the newly created address book and choose synchronize and your contacts should download.</li> </ul> <h2 id="configure-android-phone">Configure Android Phone</h2> <h3 id="clear-current-google-apps">Clear Current Google Apps</h3> <ul> <li>Go to <kbd>Application Manager</kbd> → <kbd>All</kbd></li> <li><kbd>Google Calendar Sync</kbd> → <kbd>Clear Data</kbd> → <kbd>OK</kbd> → <kbd>Disable</kbd> → <kbd>OK</kbd>.</li> <li><kbd>Google Contacts Sync</kbd> → <kbd>Clear Data</kbd> → <kbd>OK</kbd> → <kbd>Disable</kbd> → <kbd>OK</kbd>.</li> </ul> <h3 id="contacts-1">Contacts</h3> <ul> <li>Go to <kbd>People</kbd>.</li> <li>Choose <kbd>Settings</kbd> → <kbd>Accounts</kbd> → <kbd>Your Google account</kbd>.</li> <li>Clear the checkbox so that Google Contacts are no longer synchronized.</li> <li>Install CardDAV from the Google Play Store. There is a free version CardDav-Sync free beta and a paid version called CardDav-Sync beta. The paid version syncs more contact fields. You can start with the free and upgrade later.</li> <li>Run CardDAV-Sync and use these settings:</li> <li>Server URL: &lt;servername&gt;/owncloud/remote.php/carddav/ <ul> <li>Use SSL: check accordingly.</li> <li>Username: your login name.</li> <li>Password: your password.</li> </ul> </li> <li>Click <kbd>OK</kbd>.</li> <li>Go back to People.</li> <li>Choose <kbd>Settings</kbd> → <kbd>Contacts to display</kbd> → <kbd>Customize</kbd>.</li> <li>Expand your Google account and clear all of it’s checkboxes.</li> <li>Expand your CardDav account and check all of of it’s checkboxes.</li> <li>Click <kbd>OK</kbd> and your contacts will be synchronized.</li> </ul> <h3 id="calendar-1">Calendar</h3> <ul> <li>Install CalDAV from the Google Play Store. There is a free version called CalDav-Sync free beta and a paid version called CalDav-Sync beta, both written by Marten Gajda.</li> <li>Open Calendar.</li> <li>Choose <kbd>&#9881;</kbd> → <kbd>Settings</kbd> → <kbd>Your Google account</kbd>.</li> <li>Clear the checkbox so that Google Calendar is no longer synchronized.</li> <li>Click <kbd>Add Account</kbd> → <kbd>CalDav</kbd> and use these settings.</li> <li>Server URL: &lt;servername&gt;/owncloud/remote.php/caldav/ <ul> <li>Use SSL: check accordingly.</li> <li>Username: your login name.</li> <li>Password: your password.</li> <li>Click Next.</li> </ul> </li> <li>Under “Select accounts to sync” select all appropriate calendars.</li> <li>Click <kbd>Next</kbd>.</li> <li>Enter the email address. The email address is used for email notifications.</li> <li>Click OK and your calendar(s) will be synchronized.</li> <li>Go to <kbd>phone Settings</kbd> → <kbd>Accounts</kbd> → <kbd>Google</kbd> → Select the Google account with the sync icon next to it → Uncheck <kbd>calendar</kbd>.</li> </ul> <p>Congratulations, you now no longer need an external service to synchronize your contacts and calendar. As a bonus there are a bunch of ownCloud add-ons that give you much more power than you previously had.</p> Tue, 26 Mar 2013 10:42:00 +0000 http://flailingmonkey.com/moving-contacts-calendar-google/ http://flailingmonkey.com/moving-contacts-calendar-google/ How to install Django on JustHost (and maybe other shared hosts) Mike Ratcliffe <figure> <p><img src="/assets/images/frustrated-developer.jpg" alt="Frustrated developer" /></p> <figcaption>Frustrated developer</figcaption> </figure> <p>If you search Google for “how to install Django on JustHost” you may be surprised to find tens of thousands of articles claiming that it can’t be done and none claiming that it can. Well, it can and here’s how:</p> <h2 id="log-in-using-ssh-access">Log in using SSH access</h2> <ol> <li>In the Security section of your control panel click <kbd>SSH/Shell Access</kbd> → <kbd>Manage SSH Keys</kbd>.</li> <li>Click <kbd>Generate a new key</kbd>, enter a password and click <kbd>Generate Key</kbd>.</li> <li>After the key is generated click <kbd>Go back</kbd>.</li> <li>Click the <kbd>Manage Authorisation</kbd> and click <kbd>Authorize</kbd>.</li> <li>Now download the private key to the folder in which you would like to keep it.</li> <li>Add the key to your SSH Client.</li> <li>Connect to your primary domain using your JustHost login information (the same as logging in to your cPanel).</li> </ol> <h2 id="install-python-272">Install Python 2.7.2</h2> <div class="highlight"><pre><code class="language-shell">mkdir ~/python <span class="nb">cd</span> ~/python wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz tar zxfv Python-2.7.2.tgz rm -rf Python-2.7.2.tgz find ~/python -type d <span class="p">|</span> xargs chmod <span class="m">0755</span> <span class="nb">cd</span> Python-2.7.2 ./configure --prefix<span class="o">=</span><span class="nv">$HOME</span>/python make make install <span class="nb">cd</span> .. rm -rf Python-2.7.2</code></pre></div> <h2 id="modify-bashrc">Modify ~/.bashrc</h2> <div class="highlight"><pre><code class="language-shell">vim ~/.bashrc <span class="c1"># Press i and enter the following:</span> <span class="nb">export</span> <span class="nv">PATH</span><span class="o">=</span><span class="nv">$HOME</span>/python/bin:<span class="nv">$PATH</span> <span class="c1"># press &lt;escape&gt;:wq&lt;enter&gt;</span> <span class="nb">source</span> ~/.bashrc <span class="c1"># Whenever you want to work using Python 2.7 in the</span> <span class="c1"># console you will need to enter this</span></code></pre></div> <h2 id="test-your-python-install">Test your Python install</h2> <div class="highlight"><pre><code class="language-shell">python -V <span class="c1"># This should output &quot;Python 2.7.2&quot;</span></code></pre></div> <h2 id="install-setuptools">Install setuptools</h2> <div class="highlight"><pre><code class="language-shell">wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz tar xzvf setuptools-0.6c11.tar.gz rm setuptools-0.6c11.tar.gz <span class="nb">cd</span> setuptools-0.6c11 python setup.py install <span class="nb">cd</span> .. rm -rf setuptools-0.6c11</code></pre></div> <h2 id="install-pip">Install pip</h2> <div class="highlight"><pre><code class="language-shell">wget http://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz tar xzvf pip-1.2.1.tar.gz rm pip-1.2.1.tar.gz <span class="nb">cd</span> pip-1.2.1 python setup.py install <span class="nb">cd</span> .. rm -rf pip-1.2.1</code></pre></div> <h2 id="install-django-flup-and-mysql-python-module">Install Django, flup and MySQL Python Module</h2> <p>Now that pip is installed this is easy part.</p> <div class="highlight"><pre><code class="language-shell">pip install Django pip install flup pip install MySQL-python</code></pre></div> <h2 id="create-htaccess">Create .htaccess</h2> <div class="highlight"><pre><code class="language-shell"><span class="nb">cd</span> ~/public_html/mysite <span class="c1"># where mysite is the root folder of your site</span> vim .htaccess <span class="c1"># Press i then enter the following:</span> AddHandler fcgid-script .fcgi RewriteEngine On RewriteCond %<span class="o">{</span>REQUEST_FILENAME<span class="o">}</span> !-f RewriteRule ^<span class="o">(</span>.*<span class="o">)</span>$ mysite.fcgi/<span class="nv">$1</span> <span class="o">[</span>QSA,L<span class="o">]</span> <span class="c1"># Press &lt;escape&gt;:wq&lt;enter&gt;</span></code></pre></div> <h2 id="configure-your-website">Configure Your Website</h2> <div class="highlight"><pre><code class="language-shell">mkdir ~/django-projects <span class="nb">cd</span> ~/django-projects django-admin.py startproject myproject <span class="c1"># I use the same name as mysite</span> <span class="c1"># without any periods</span> <span class="nb">cd</span> ~/public_html/mysite vim mysite.fcgi <span class="c1"># Press i then enter the following:</span> <span class="c1">#!/home/your_username/python/bin/python</span> import sys, os <span class="c1"># Where /home/your_username is the path to your home directory</span> sys.path.insert<span class="o">(</span><span class="m">0</span>, <span class="s2">&quot;/home/your_username/python&quot;</span><span class="o">)</span> sys.path.insert<span class="o">(</span><span class="m">13</span>, <span class="s2">&quot;/home/your_username/django-projects/myproject&quot;</span><span class="o">)</span> os.environ<span class="o">[</span><span class="s1">&#39;DJANGO_SETTINGS_MODULE&#39;</span><span class="o">]</span> <span class="o">=</span> <span class="s1">&#39;myproject.settings&#39;</span> from django.core.servers.fastcgi import runfastcgi runfastcgi<span class="o">(</span><span class="nv">method</span><span class="o">=</span><span class="s2">&quot;threaded&quot;</span>, <span class="nv">daemonize</span><span class="o">=</span><span class="s2">&quot;false&quot;</span><span class="o">)</span> <span class="c1"># Press &lt;escape&gt;:wq&lt;enter&gt;</span> chmod <span class="m">0755</span> mysite.fcgi</code></pre></div> <h2 id="test-your-configuration">Test your configuration</h2> <div class="highlight"><pre><code class="language-shell">python mysite.fcgi <span class="c1"># Fix any errors and run the command again. When you have an HTML page</span> <span class="c1"># returned your Django install is complete. You should be able to go to your</span> <span class="c1"># sites URL and view the Django welcome page.</span></code></pre></div> <h2 id="thats-everything">That’s everything</h2> <p>Now you can get to work and create some real Django pages.</p> <h2 id="a-few-tips">A Few Tips</h2> <ul> <li>Remember to install every python and django package using pip - that way they will be available to your django app.</li> <li>Keep in mind that every Django command like syncdb, collectstatic  etc. should be run using the correct version of Python. This means using source ~/.bashrc every time you log into your SSH client e.g. <div class="highlight"><pre><code class="language-javascript"><span class="nx">source</span> <span class="o">~</span><span class="err">/.bashrc</span> <span class="nx">python</span> <span class="nx">manage</span><span class="p">.</span><span class="nx">py</span> <span class="nx">syncdb</span> <span class="nx">python</span> <span class="nx">manage</span><span class="p">.</span><span class="nx">py</span> <span class="nx">shell</span> <span class="nx">etc</span><span class="p">.</span></code></pre></div> </li> </ul> Mon, 21 Jan 2013 12:00:00 +0000 http://flailingmonkey.com/install-django-justhost/ http://flailingmonkey.com/install-django-justhost/ Firefox DevTools API Mike Ratcliffe <p>Hopefully, most of you are familiar with <a href="https://developer.mozilla.org/docs/Tools" title="Firefox Developer Tools">Firefox Developer Tools</a> by now, if not then now is the time to get familiar! The DevTools Team recently met up in London to talk about, and generally hack on, developer tools stuff and it turned out to be an amazingly productive time. The DevTools team has two branches, the debugger team and the inspector team so I will discuss them separately.</p> <h2 id="the-debugger-team-aka-team-minotaur">The Debugger Team a.k.a. Team Minotaur</h2> <p>Rob Campbell talked about how to build a Jetpack addon using Scratchpad. They have managed to get <a href="http://blog.astithas.com/2012/10/debugging-firefox.html" title="Firefox chrome debugging">Firefox chrome debugging</a> working… it is still work in progress but it will be extremely useful for debugging Firefox OS etc. Mihai Sucan managed to get the Web Console working to log errors, network requests etc from a <a href="https://wiki.mozilla.org/Gaia" title="Gaia">Gaia</a> installation. They even managed to step through CoffeeScript using sourcemaps!</p> <p>Not debugger related but also by a member of that team, Victor Porof demoed WebGL live shader editing using scratchpad.</p> <h2 id="the-inspector-team-aka-team-carnage-">The Inspector Team a.k.a. Team Carnage <sup id="fnref:carnage"><a href="#fn:carnage" class="footnote">1</a></sup></h2> <p>Paul Rouget really was on fire, he has managed to get the Responsive Design View to use mobile style scrollbars instead of the standard ones.</p> <iframe src="http://www.youtube.com/embed/M2y5S7p81jU?fs=1" allowfullscreen=""></iframe> <p>Paul has also managed to add a live preview to the DOM Tree à la Sublime Text and added an OSX fullscreen button to the scratchpad.</p> <iframe src="http://www.youtube.com/embed/61tQij_ox9k?fs=1" allowfullscreen=""></iframe> <p>Paul, Heather and myself managed to get a new panel management system working. These panels will be a little like Firebug panels on steroids and will ultimately make all of our lives much easier, more about that below.</p> <h2 id="others">Others</h2> <p>Mark Goodwin has rewritten the <a href="https://developer.mozilla.org/docs/Tools/GCLI" title="The Firefox Command Line">Command Line</a> cookie command so that it is much more reliable when creating and editing cookies. Kevin Dangoor added an in page HTML editor that can be used along with the DOM Tree for Dreamweaver-like editing.</p> <h2 id="the-devtools-api">The DevTools API</h2> <p>Before I say anything about this API I would like to add that it is so early alpha that it is a long way away from being merged with Firefox’s main branches. Everything here is subject to change so don’t use it, at least no until we have everything working properly! Paul Rouget has also blogged about the <a href="http://paulrouget.com/e/devtools-toolbox/" title="The DevTools Toolbox">DevTools Toolbox</a> in a little less detail. I stole the images on this page from him to save a little time… thanks Paul.</p> <p>I mentioned earlier that we will be moving to panels similar to Firebug’s. Of course, a lot of people like our sidebar so we didn’t want to lose that. Unfortunately, if every tool was allowed to choose how and where it’s panels are opened we very quickly have no space for the content area, we call this phenomena “The DevTools Donut.”</p> <figure> <p><img src="/assets/images/firefox-developer-tools-donut.jpg" alt="Firefox Developer Tools Donut" /></p> <figcaption>Firefox developer tools donut</figcaption> </figure> <p>We have also received complaints that opening a sidebar causes reflow and can make it impossible for pages that use media queries to be edited properly. Others do not like editing at the bottom of the screen like in Firebug because some tools are better off in a side panel.</p> <p>The DevTools API allows a tool to register itself. On registration the tool will be passed an iframe in which it will be responsible for rendering itself. Of course, we will send events to the tool so that it can react as appropriate. This means that tools can forget about panels and leave all panel related activities to be handled by the panel manager.</p> <p>Registering a tool is fairly simple:</p> <div class="highlight"><pre><code class="language-js"><span class="nx">gDevTools</span><span class="p">.</span><span class="nx">registerTool</span><span class="p">({</span> <span class="nx">id</span><span class="o">:</span> <span class="s2">&quot;myToolId&quot;</span><span class="p">,</span> <span class="nx">killswitch</span><span class="o">:</span> <span class="s2">&quot;devtools.mytool.enabled&quot;</span><span class="p">,</span> <span class="c1">// Your tool needs to create this</span> <span class="nx">icon</span><span class="o">:</span> <span class="s2">&quot;chrome://browser/skin/devtools/mytool-tool-icon.png&quot;</span><span class="p">,</span> <span class="nx">url</span><span class="o">:</span> <span class="s2">&quot;chrome://browser/content/devtools/mytool.xul&quot;</span><span class="p">,</span> <span class="nx">label</span><span class="o">:</span> <span class="nx">l10n</span><span class="p">.</span><span class="nx">getStr</span><span class="p">(</span><span class="s2">&quot;ToolboxMyTool.label&quot;</span><span class="p">),</span> <span class="nx">build</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">aIFrameWindow</span><span class="p">,</span> <span class="nx">aTarget</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="k">new</span> <span class="nx">MyToolInstance</span><span class="p">(</span><span class="nx">aIFrameWindow</span><span class="p">,</span> <span class="nx">aTarget</span><span class="p">);</span> <span class="p">}</span> <span class="p">});</span></code></pre></div> <p>MyToolInstance needs to have a build method that accepts two arguments, an iframe.contentWindow and a target. The window is the window in which your tool is to build itself and the target is an object that specifies a gDevTools.TargetType and a value such as a tab. It also needs a destroy method so that it can clean up listeners etc. when the toolbox is closed.</p> <p>The tools open in what we call the Toolbox. This toolbox appears at the bottom of the content area by default but can also be moved to the right of the content area or popped out. Tools will keep their state when changing tabs so you do not usually need any special mechanism to save and restore state.</p> <figure> <p><img src="/assets/images/toolbox-displayed-below-content.jpg" alt="Toolbox displayed below content" /></p> <figcaption>Toolbox displayed below content</figcaption> </figure> <figure> <p><img src="/assets/images/toolbox-displayed-on-the-right-of-content.jpg" alt="Toolbox displayed on the right of content" /></p> <figcaption>Toolbox displayed on the right of content</figcaption> </figure> <figure> <p><img src="/assets/images/toolbox-undocked.jpg" alt="Toolbox undocked" /></p> <figcaption>Toolbox undocked</figcaption> </figure> <p>Tools do not need to live in a panel. We also have a mechanism that allows you to register a Firefox Command Line command and display it as a button. This makes it extremely easy to add buttons to create screenshots, trigger responsive design mode, restart Firefox etc.</p> <p>Are we missing anything? I don’t think so but if we are then feel free to let us know.</p> <h3 id="resources">Resources</h3> <div class="footnotes"> <ol> <li id="fn:carnage"> <p>I can neither confirm or deny that “Team Carnage” is the inspector team’s official name although we may or may not be searching for one. If you can think of a better name then please let us know. <a href="#fnref:carnage" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Fri, 05 Oct 2012 09:32:00 +0100 http://flailingmonkey.com/firefox-devtools-api/ http://flailingmonkey.com/firefox-devtools-api/ How well does your browser protect your privacy? Mike Ratcliffe <p>Having downsized my home a couple of times in the last year it is no understatement to say I have urgent space issues. I have power tools, large kitchen devices and surplus everything else. I looked online for a new shed, comparing reviews and investigating the UK’s top shed building companies in my typically obsessive way. Having found “the one”, I paid the company and am currently waiting for the timber that I will use to make the concrete base for the shed in preparation for delivery.</p> <p>Imagine my irritation a few days after my investigations to discover that almost every web page I visit contains some kind of shed advert! It seems that the popular search engine that I use actually owns a large advertising and tracking company and that this company advertises on a very large percentage of websites. A lot has been said about online privacy and users rights to own their data but little has been said about what can be done about it.</p> <p>It irks me that my search results have suddenly become shed based… wood, cedar, paint, tools internet and house are all terms that result in the first few results being related to sheds. Interestingly there is a “Why do I see this?” link next to these results and clicking this link provides an option not to receive these targeted results. Conveniently, if I attempt to opt out nothing changes and if I look at the option again it remains set for me to receive targeted results. I also find that the search results that I receive appear to be the results that the search company think that I want to receive.</p> <p>I greatly dislike the fact that if you and I search for a particular term both us us will receive a completely different set of results. I would love to say that tracking a user in this way should be illegal but the truth is that online advertising is very big business and it won’t be going away any time soon. A quick look at the profits of large search engine and social media companies shows just how lucrative online advertising can be. I have nothing against online advertising itself but I do have a problem with the methods that some websites and online stores employ in order to track my online activity.</p> <p>Before the <a href="http://edition.cnn.com/2012/03/01/tech/social-media/twitter-selling-tweets/">recent(ish) media outrage</a> would you ever have thought that Twitter would begin selling it’s users tweets? Google’s mission is to organize the world’s information and make it universally accessible and useful. Lots of people I know find this a frightening concept. Need I mention Facebook and the many privacy issues that have been reported. Personally, I don’t mind the idea of them using e.g. pictures that I have uploaded, after all, I did upload the picture to their servers. I really do dislike them using my data to target me with their adverts.</p> <p>Tracking aside, there are bigger issues to worry about. Many web sites, in particularly forums, send usernames and passwords in plain text. When the data is sent over the internet it goes through a bunch of servers before it arrives at it’s destination. It is simple to find the path that your data takes using traceroute:</p> <div class="highlight"><pre><code class="language-powershell"><span class="n">tracert</span> <span class="n">google</span><span class="p">.</span><span class="n">com</span> <span class="n">Tracing</span> <span class="n">route</span> <span class="n">to</span> <span class="n">google</span><span class="p">.</span><span class="n">com</span> <span class="p">[</span><span class="n">173</span><span class="p">.</span><span class="n">194</span><span class="p">.</span><span class="n">34</span><span class="p">.</span><span class="n">72</span><span class="p">]</span> <span class="n">over</span> <span class="n">a</span> <span class="n">maximum</span> <span class="n">of</span> <span class="n">30</span> <span class="n">hops</span><span class="err">:</span> <span class="n">1</span> <span class="p">&lt;</span><span class="n">1</span> <span class="n">ms</span> <span class="p">&lt;</span><span class="n">1</span> <span class="n">ms</span> <span class="p">&lt;</span><span class="n">1</span> <span class="n">ms</span> <span class="n">my</span><span class="p">.</span><span class="n">router</span> <span class="p">[</span><span class="n">192</span><span class="p">.</span><span class="n">168</span><span class="p">.</span><span class="n">1</span><span class="p">.</span><span class="n">1</span><span class="p">]</span> <span class="n">2</span> <span class="n">24</span> <span class="n">ms</span> <span class="n">4</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">93</span><span class="p">.</span><span class="n">186</span><span class="p">.</span><span class="n">155</span><span class="p">.</span><span class="n">65-ifnl</span><span class="p">.</span><span class="n">net</span> <span class="p">[</span><span class="n">93</span><span class="p">.</span><span class="n">186</span><span class="p">.</span><span class="n">155</span><span class="p">.</span><span class="n">65</span><span class="p">]</span> <span class="n">3</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">ge</span><span class="p">-</span><span class="n">2</span><span class="p">-</span><span class="n">0</span><span class="p">-</span><span class="n">0-xcr1</span><span class="p">.</span><span class="n">lsw</span><span class="p">.</span><span class="n">cw</span><span class="p">.</span><span class="n">net</span> <span class="p">[</span><span class="n">194</span><span class="p">.</span><span class="n">112</span><span class="p">.</span><span class="n">69</span><span class="p">.</span><span class="n">129</span><span class="p">]</span> <span class="n">4</span> <span class="n">6</span> <span class="n">ms</span> <span class="n">4</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">xe</span><span class="p">-</span><span class="n">4</span><span class="p">-</span><span class="n">3</span><span class="p">-</span><span class="n">0-xcr1</span><span class="p">.</span><span class="n">lns</span><span class="p">.</span><span class="n">cw</span><span class="p">.</span><span class="n">net</span> <span class="p">[</span><span class="n">195</span><span class="p">.</span><span class="n">2</span><span class="p">.</span><span class="n">28</span><span class="p">.</span><span class="n">42</span><span class="p">]</span> <span class="n">5</span> <span class="n">81</span> <span class="n">ms</span> <span class="n">85</span> <span class="n">ms</span> <span class="n">82</span> <span class="n">ms</span> <span class="n">72</span><span class="p">.</span><span class="n">14</span><span class="p">.</span><span class="n">198</span><span class="p">.</span><span class="n">113</span> <span class="n">6</span> <span class="n">36</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">6</span> <span class="n">ms</span> <span class="n">209</span><span class="p">.</span><span class="n">85</span><span class="p">.</span><span class="n">240</span><span class="p">.</span><span class="n">63</span> <span class="n">7</span> <span class="n">6</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">209</span><span class="p">.</span><span class="n">85</span><span class="p">.</span><span class="n">253</span><span class="p">.</span><span class="n">31</span> <span class="n">8</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">5</span> <span class="n">ms</span> <span class="n">lhr14s19-in-f8</span><span class="p">.</span><span class="n">1e100</span><span class="p">.</span><span class="n">net</span> <span class="p">[</span><span class="n">173</span><span class="p">.</span><span class="n">194</span><span class="p">.</span><span class="n">34</span><span class="p">.</span><span class="n">72</span><span class="p">]</span></code></pre></div> <p>All eight steps in this traceroute are points where a device could be intercepting your usernames and passwords and you would know nothing about it. One tool that helps protect against this kind of activity is the Secure Sockets Layer (SSL).</p> <p>SSL provides a way to create an encrypted channel between an application running on your computer and an application running on another computer. When it comes to web pages then the secure protocol that we use is HTTPS. When you are on a webpage containing a log in dialog it is only safe to submit your username and password only if the page is an HTTPS page.</p> <p>So, why don’t we just use HTTPS all of the time, would that not solve all of our privacy issues? No, unfortunately not. HTTPS ensures that your data is safe between your computer and the destination server. Once your data is on the server it is in the hands of the organization that owns the server. This is highly likely to be a legitimate server but it could just as easily be an evil webmaster that is attempting to gather your personal information.</p> <p>A useful Firefox extension that will use HTTPS versions of a site whenever available is <a href="https://www.eff.org/https-everywhere" title="HTTP Anywhere">HTTPS Everywhere</a>. I would certainly suggest using it, after it is installed you will find that a large amount of sites will automatically run using HTTPS.</p> <p>Another issue that plagues the internet today is website spoofing / phishing. Website spoofers create websites that looks very similar to a site that you commonly use with the soul purpose of gaining information about you. In the last couple of years sites that directly ask you for information such as bank or credit card details have become more popular. An example of this kind of site can be found <a href="http://ismycreditcardstolen.com" title="Is my credit card stolen">here</a>. The page does not submit any information, in fact you don’t really need to enter anything, it is designed as a teaching aid. Try it, you can trust it… honest (to be safe don’t use your real information).</p> <p>Compared to the illegal vulnerabilities that we face, tracking is not such a worry but more of an irritating abuse of privacy. Google, Facebook, Twitter and others make money by providing users with a free service in return for information about them. This information is either used by the sites themselves or sold to other sites in order to better target you with adverts that they believe that you will respond to. This video highlights this issue in quite an eye opening way:</p> <iframe src="http://www.youtube.com/embed/yZKul_awxMA" allowfullscreen=""></iframe> <p>Google and Facebook have built relationships with millions of other websites to help them track your web activity using cookies. Google uses it’s adverts, and analytical services to track what you do whilst Facebook uses it’s log in services and like buttons. It is possible to disable cookies in your browser preferences but you would find that many sites would not work properly and you would have to log in every time you visit. Deleting cookies can help but some websites create zombie cookies (the word cookie here is misleading).</p> <p>Zombie cookies are stored in multiple locations using multiple methods e.g. Flash LSOs, cookies, local storage, web databases etc. When a zombie cookie is erased it recreates itself along with it’s gathered information when you return to a related website.</p> <p>There is an excellent Firefox extension called <a href="https://addons.mozilla.org/firefox/addon/lightbeam/">Lightbeam</a>. Lightbeam allows you to see what kind of communications go on between the sites you visit. When the extension is installed click on the white circle at the bottom right of the browser to start. A tree similar to the one below will appear with each node representing a website or domain. The collusion developers are working on adding more features, such as the ability to click on any node in the graph and tell Firefox to block third-party cookies to that site. They are also working on visualizing other methods of tracking besides third-party cookies. If you would like to keep an eye on their progress, file a bug, request a feature, or get a copy of the source code, please go to <a href="http://github.com/toolness/collusion">http://github.com/toolness/collusion</a>.</p> <figure> <p><img src="/assets/images/the-excellent-collusion-extension.jpg" alt="The excellent Lightbeam extension" /></p> <figcaption>The excellent Lightbeam extension</figcaption> </figure> <p>Unfortunately, even when cookies are blocked there are things that websites can do in order to track you. Take for example a Facebook like button, even without cookies it would be capable of informing Facebook which website you were on when you clicked the button and which page referred you to that website. There is really very little, if anything, that you can do in order to prevent this.</p> <p>Another technique that can be used to identify you is fingerprinting. This is a method of gathering information about your browser in an attempt to uniquely identify you. Although you may think your browser is not unique consider how unique your personal combination of screen size, color depth, time zone, user agent and browser plugins. You can check how unique your browser is <a href="http://panopticlick.eff.org/" title="How unique is your browser">here</a>, the odds are that you can be uniquely identified. This method could potentially be used to track you without using cookies but at the time of writing it is not commonly used.</p> <p>I have always worried about my online privacy. In fact, it is because Mozilla is a non profit organization that I work for them. They are not an advertising company so I can be absolutely certain that my browser is not spying on me. Of course, tracking cookies are part of the web so we can’t block them but we are actively working on ways to give you control of this area of the internet. Another video:</p> <iframe src="http://www.youtube.com/embed/2J0ITwUZ4lc" allowfullscreen=""></iframe> <p>Cheesy maybe, but very educational.</p> <p>Firefox contains “<a href="https://www.mozilla.org/dnt/">Do not track</a>.” Do Not Track is a feature in Firefox that allows you to let a website know you would like to opt-out of third-party tracking for purposes including behavioral advertising. Companies are starting to support Do Not Track, but you may not notice many changes until more tracking companies choose to honor it. Mozilla are actively working with companies that have started to implement Do Not Track and with others who have committed to doing so soon. How do you enable “Do not track?” Simple, watch this video:</p> <iframe src="http://www.youtube.com/embed/D0ieagw4jFI" allowfullscreen=""></iframe> <p>In fact, if you want to go all out against tracking and advertising you could also install these Firefox extensions:</p> <ul> <li><a href="https://www.eff.org/https-everywhere">HTTPS Everywhere</a> - As mentioned above, HTTPS Everywhere is a Firefox and Chrome extension that encrypts your communications with many major websites, making your browsing more secure. It is available for Firefox and Chrome bug due to a Chrome bug has far better stability in Firefox.</li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/?src=cb-dl-users">Adblock Plus</a> - Annoyed by adverts? Troubled by tracking? Bothered by banners? Install Adblock Plus now to regain control of the internet and change the way that you view the web.</li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/betterprivacy/?src=cb-dl-users">Better Privacy</a> - Remove or manage a new and uncommon kind of cookies, better known as LSO’s (mentioned above when talking about zombie cookies). The BetterPrivacy safeguard offers various ways to handle Flash-cookies set by Google, YouTube, Ebay and others …</li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/ghostery/?src=cb-dl-users">Ghostery</a> - Ghostery sees the “invisible” web, detecting trackers, web bugs, pixels, and beacons placed on web pages by Facebook, Google Analytics, and over 1,000 other ad networks, behavioral data providers, web publishers - all companies interested in your activity.</li> <li><a href="https://addons.mozilla.org/firefox/addon/lightbeam/">Lightbeam</a> - As mentioned above, visualize who’s tracking you in real time. To get started, click on the Collusion icon in the bottom-right corner of your browser. (You may need to show the Add-on Bar to see the icon).</li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/noscript/">NoScript</a> - NoScript allows you more control over JavaScript running in your browser. It allows granular control over which scripts you would like to allow to run on any page allowing you to e.g. block JavaScript from third party sites. It really is a very good extension, but it needs to be used with care as this extension is the most likely out of this list to break websites.</li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/cookie-monster/">Cookie Monster</a> - Provides an easy NoScript-style toolbar button for changing domain-specific cookie settings so you can default to forcing session cookies and easily allow them for just the domains where you want “Remember Me” to work.</li> </ul> <p>In fact, there were 798 Firefox extensions related to Privacy &amp; Security at the time of writing, the four I have selected here are, in my opinion, the cream of the crop. Of course, you may have your own favorites… feel free to let us know.</p> Sun, 05 Aug 2012 09:22:00 +0100 http://flailingmonkey.com/browser-protect-privacy/ http://flailingmonkey.com/browser-protect-privacy/ What is it like to work as an engineer at Mozilla? Mike Ratcliffe <p>This question <a href="http://www.quora.com/Mozilla/What-is-it-like-to-work-as-an-engineer-at-Mozilla">popped up on Quora recently</a>. I have been working on the <a href="https://www.mozilla.org/en-US/firefox/technology/#tools">Developer Tools Team</a> at Mozilla for over a year now and would love to let you all know what is it like. To see this from my perspective I you will need to know a little about my background.</p> <p>I have always enjoyed reverse engineering software just to see how it ticks. If you were in the habit of using pokes to gain invulnerability or infinite lives on a Commodore or Acorn computer then I probably helped you cheat. I used to obtain many of my games from magazines because they would send them to me so that I could send them pokes or explain how to complete the latest adventure games.</p> <p>A few years later Netscape Communicator 4.0 was released and I was looking at the source. I discovered the <a href="https://www.mozilla.org/book/">about:mozilla page</a> and was impressed by the attitude that they had to open source… after all, the web is the ultimate open platform. I desperately wanted to work for Mozilla but, unfortunately, I believed that Mozilla was an organization full of Unix bearded academics each trying to show how clever they were. I am not sure where that image came from, but it was very common at the time.</p> <p>Fast forward 5 years and I was regularly contributing add-ons and fixes to the <a href="http://www.oscommerce.com/">osCommerce</a> project. I had an online store and as I tweaked my store I shared my changes with the community. I really enjoyed the fact that I was helping make tens of thousands of online stores better every time I submitted a small fix or created an add-on. I made my money working as an admin for over 60,000 users covering hundreds of domains.</p> <p>Fast forward 10 - 15 years and I had a problem with <a href="https://getfirebug.com/">Firebug</a>. I contacted Joe Hewitt and he said he would take a look. Weeks went by and, not being the patient type, I decided I would take a look myself. I fixed the problem and let Joe know about it… at this point I don’t think that the <a href="http://code.google.com/p/fbug/issues/list">Firebug Issue List</a> existed but I was surprised how quickly it was fixed in the release version. From that point I contributed Firebug fixes on and off for a few years.</p> <p>Eventually <a href="http://antennasoft.net/robcee/">Rob Campbell </a>started to hint that I should start working for Mozilla. Unfortunately, I still felt under-qualified… I was never very academic being able to learn things very quickly for myself but switching off the moment a professor starts a lecture. My <a href="https://en.wikipedia.org/wiki/Characteristics_of_dyslexia">dyslexia</a> had a role to play in that, and besides, I didn’t have a UNIX beard.</p> <figure> <p><img src="/assets/images/unix-beard.jpg" alt="UNIX Beard" /></p> <figcaption>Unix Beard</figcaption> </figure> <p>A year or two later the small company I was working for had cash flow issues and as I was the only developer working full time so I was sacrificed to the financial gods in hope of helping keep the company buoyant. Rob asked me to come to work at Mozilla again and I thought “I have been hacking since I was a kid so why the heck not?” Six months later and a number of rejected job offers later I started work with Mozilla.</p> <p>What a fantastic choice it was… Mozilla is by far the most awesome company in the world, no other company has quite the philosophy about openness as I do and to my astonishment everybody there has the same attitude.</p> <p>There are at least a gazillion reasons why I love working for Mozilla. I could list hundreds but if I had to pick a few.</p> <h2 id="positives">Positives</h2> <ol> <li>There are some people with Unix beards but they are not academic in the way I had feared. Some academics have an encyclopaedic knowledge of memorized facts and really struggle to think for themselves… well, to think outside the box. It turns out that the people here are some of the most creative and intelligent people that I have met in my life.</li> <li>I really like the fact that Mozilla’s Mission Statement is something that they believe in and live by. There are too many companies that say one thing and do another… not Mozilla.</li> <li>The people I work with truly are the cream of the crop. They are motivated and highly determined to make the Internet a better place. Everything they do every day helps them make it so. They are super smart and always happy to help anybody that has problems.</li> <li>Although each team works on separate parts of the browser we are free to fix problems as we find them. If I find a part of the browser that is not behaving properly then I can fix it without upsetting anybody, everybody is very flexible.</li> <li>In the last year the number of Mozilla employees has increased dramatically which has resulted in an incredible increase in creativity and output. It is incredibly humbling to be part of it… a day never goes by when I don’t learn something important.</li> <li>The culture at Mozilla is amazing. Before I started working here I didn’t really know anybody that thought the same way about open source as I do. Now I know hundreds of people that have the same drive and determination when it comes to openness as I do.</li> <li>Mozilla have a unique position in the browser market. They don’t need to spy on you or buy advertising and tracking companies as they are not in the advertising market. In fact, we would love to prevent the gathering of user’s browsing activities. Mozilla have no secret agenda and live by <a href="https://www.mozilla.org/about/mission.html">their mission statement</a> - to make the internet a better place.</li> <li>You know that guy at your company that you don’t like? Mozilla doesn’t have anyone like that! Seriously, I have never met a Mozilla employee that I don’t like.</li> </ol> <h2 id="negatives">Negatives</h2> <ol> <li>Because I work from home I sometimes don’t shower until late afternoon.</li> </ol> <p>That is about it for negatives, there are not many places I have worked where I can say that!</p> Fri, 03 Aug 2012 09:13:00 +0100 http://flailingmonkey.com/what-is-it-like-to-work-as-an-engineer-at-mozilla/ http://flailingmonkey.com/what-is-it-like-to-work-as-an-engineer-at-mozilla/ What is SOPA and why should you care? Mike Ratcliffe <p>I have received a flood of emails today from people asking about what SOPA is and was surprised when I looked online and didn’t see any places that explains why SOPA would be a very bad thing. The idea behind SOPA is to stop online piracy and this is a good thing but there are some very, very serious consequences meaning the end of the internet as you know it.</p> <h2 id="things-that-break-sopas-rules">Things that break SOPA’s rules</h2> <ul> <li>Any site that allows users type or upload stuff could be taken down if a user types or uploads the wrong thing.</li> <li>Uploading home movies with TV, music, computer game or even a physical DVD visible in the background… you singing a song would not be allowed. In fact, if a cereal box is in the background it would be against the rules.</li> <li>A picture of a famous person (could even be a poster on a wall that is part of some photo or video) would also be a reason to take down a site.</li> </ul> <p>Any site breaking these rules would be taken down.</p> <h2 id="the-effect-that-this-would-have">The effect that this would have</h2> <ul> <li>Site owners will not only be responsible for their own comments and pictures but they will become responsible for comments and pictures that users upload in their comments. This means that every site owner will need to review every users comments, pictures, videos, etc. to ensure that they do not break any rules.</li> <li>Facebook, YouTube, Wikipedia, blogs etc. could all become a thing of the past as e.g. Twitter cannot be expected to review over 10,000 comments per minute and YouTube cannot be expected to check 8 years of content per day to see if they break any rules.</li> <li>Book reviews, music reviews etc. would become a thing of the past as having one of those things on your site would be against the rules.</li> <li>There would be almost no cool new sites and the sites that you know and love would be taken down.</li> <li>Basically any site that you can imagine that allows you to upload or type stuff would be taken down!</li> </ul> <p>Khan Academy have done a great job of explaining the details in this video:</p> <iframe src="http://www.youtube.com/embed/OjAb5zPqAlg?fs=1" allowfullscreen=""></iframe> <p>Of course there are more side effects but even the small amount of information above should help you see how huge an issue this is. If you live in the USA then you can sign a petition to stop this <a href="http://sopastrike.com/modal/strike-modal/index.html">here</a>.</p> Wed, 18 Jan 2012 23:10:00 +0000 http://flailingmonkey.com/what-is-sopa-and-why-should-you-care/ http://flailingmonkey.com/what-is-sopa-and-why-should-you-care/ How Browsers Work Mike Ratcliffe <figure> <p><img src="/assets/images/safari-firefox-and-chrome-icons.jpg" alt="Safari Firefox and Chrome Icons" /></p> </figure> <ul id="markdown-toc"> <li><a href="#preface" id="markdown-toc-preface">Preface</a></li> <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a> <ul> <li><a href="#the-browsers-we-will-talk-about" id="markdown-toc-the-browsers-we-will-talk-about">The Browsers We Will Talk About</a></li> <li><a href="#the-browsers-main-functionality" id="markdown-toc-the-browsers-main-functionality">The Browser’s Main Functionality</a></li> <li><a href="#the-browsers-high-level-structure" id="markdown-toc-the-browsers-high-level-structure">The Browser’s High Level Structure</a></li> <li><a href="#communication-between-the-components" id="markdown-toc-communication-between-the-components">Communication Between the Components</a></li> </ul> </li> <li><a href="#the-rendering-engine" id="markdown-toc-the-rendering-engine">The Rendering Engine</a> <ul> <li><a href="#rendering-engines" id="markdown-toc-rendering-engines">Rendering Engines</a></li> <li><a href="#the-main-flow" id="markdown-toc-the-main-flow">The Main Flow</a></li> <li><a href="#main-flow-examples" id="markdown-toc-main-flow-examples">Main Flow Examples</a></li> <li><a href="#parsing-and-dom-tree-construction" id="markdown-toc-parsing-and-dom-tree-construction">Parsing and DOM Tree Construction</a> <ul> <li><a href="#parsing---general" id="markdown-toc-parsing---general">Parsing - General</a> <ul> <li><a href="#grammars" id="markdown-toc-grammars">Grammars</a></li> <li><a href="#parser---lexer-combination" id="markdown-toc-parser---lexer-combination">Parser - Lexer Combination</a></li> <li><a href="#translation" id="markdown-toc-translation">Translation</a></li> <li><a href="#parsing-example" id="markdown-toc-parsing-example">Parsing Example</a></li> <li><a href="#formal-definitions-for-vocabulary-and-syntax" id="markdown-toc-formal-definitions-for-vocabulary-and-syntax">Formal Definitions for Vocabulary and Syntax</a></li> <li><a href="#types-of-parsers" id="markdown-toc-types-of-parsers">Types of Parsers</a></li> <li><a href="#generating-parsers-automatically" id="markdown-toc-generating-parsers-automatically">Generating Parsers Automatically</a></li> </ul> </li> <li><a href="#html-parser" id="markdown-toc-html-parser">HTML Parser</a> <ul> <li><a href="#the-html-grammar-definition" id="markdown-toc-the-html-grammar-definition">The HTML Grammar Definition</a></li> <li><a href="#not-a-context-free-grammar" id="markdown-toc-not-a-context-free-grammar">Not a Context Free Grammar</a></li> <li><a href="#html-dtd" id="markdown-toc-html-dtd">HTML DTD</a></li> <li><a href="#dom" id="markdown-toc-dom">DOM</a></li> <li><a href="#the-parsing-algorithm" id="markdown-toc-the-parsing-algorithm">The Parsing Algorithm</a></li> <li><a href="#the-tokenization-algorithm" id="markdown-toc-the-tokenization-algorithm">The Tokenization Algorithm</a></li> <li><a href="#tree-construction-algorithm" id="markdown-toc-tree-construction-algorithm">Tree Construction Algorithm</a></li> <li><a href="#actions-when-the-parsing-is-finished" id="markdown-toc-actions-when-the-parsing-is-finished">Actions When the Parsing is Finished</a></li> <li><a href="#browsers-error-tolerance" id="markdown-toc-browsers-error-tolerance">Browsers Error Tolerance</a> <ul> <li><a href="#br-instead-of-br" id="markdown-toc-br-instead-of-br">&lt;/br&gt; Instead of &lt;br&gt;</a></li> <li><a href="#a-stray-table" id="markdown-toc-a-stray-table">A Stray Table</a></li> <li><a href="#nested-form-elements" id="markdown-toc-nested-form-elements">Nested Form Elements</a></li> <li><a href="#a-too-deep-tag-hierarchy" id="markdown-toc-a-too-deep-tag-hierarchy">A Too Deep Tag Hierarchy</a></li> <li><a href="#misplaced-html-or-body-end-tags" id="markdown-toc-misplaced-html-or-body-end-tags">Misplaced html or body End Tags</a></li> </ul> </li> </ul> </li> <li><a href="#css-parsing" id="markdown-toc-css-parsing">CSS Parsing</a> <ul> <li><a href="#webkit-css-parser" id="markdown-toc-webkit-css-parser">Webkit CSS Parser</a></li> </ul> </li> <li><a href="#parsing-scripts" id="markdown-toc-parsing-scripts">Parsing Scripts</a></li> <li><a href="#the-order-of-processing-scripts-and-stylesheets" id="markdown-toc-the-order-of-processing-scripts-and-stylesheets">The Order of Processing Scripts and Stylesheets</a> <ul> <li><a href="#scripts" id="markdown-toc-scripts">Scripts</a></li> <li><a href="#speculative-parsing" id="markdown-toc-speculative-parsing">Speculative Parsing</a></li> <li><a href="#stylesheets" id="markdown-toc-stylesheets">Stylesheets</a></li> </ul> </li> </ul> </li> <li><a href="#render-tree-construction" id="markdown-toc-render-tree-construction">Render Tree Construction</a> <ul> <li><a href="#the-render-tree-relation-to-the-dom-tree" id="markdown-toc-the-render-tree-relation-to-the-dom-tree">The Render Tree Relation to the DOM Tree</a></li> <li><a href="#the-flow-of-constructing-the-tree" id="markdown-toc-the-flow-of-constructing-the-tree">The Flow of Constructing the Tree</a></li> <li><a href="#style-computation" id="markdown-toc-style-computation">Style Computation</a> <ul> <li><a href="#sharing-style-data" id="markdown-toc-sharing-style-data">Sharing Style Data</a></li> <li><a href="#firefox-rule-tree" id="markdown-toc-firefox-rule-tree">Firefox Rule Tree</a> <ul> <li><a href="#division-into-structs" id="markdown-toc-division-into-structs">Division Into Structs</a></li> <li><a href="#computing-the-style-contexts-using-the-rule-tree" id="markdown-toc-computing-the-style-contexts-using-the-rule-tree">Computing the Style Contexts Using the Rule Tree</a></li> </ul> </li> <li><a href="#manipulating-the-rules-for-an-easy-match" id="markdown-toc-manipulating-the-rules-for-an-easy-match">Manipulating the Rules for an Easy Match</a></li> <li><a href="#applying-the-rules-in-the-correct-cascade-order" id="markdown-toc-applying-the-rules-in-the-correct-cascade-order">Applying the Rules in the Correct Cascade Order</a> <ul> <li><a href="#stylesheet-cascade-order" id="markdown-toc-stylesheet-cascade-order">Stylesheet Cascade Order</a></li> <li><a href="#specificity" id="markdown-toc-specificity">Specificity</a></li> <li><a href="#sorting-the-rules" id="markdown-toc-sorting-the-rules">Sorting the rules</a></li> </ul> </li> </ul> </li> <li><a href="#gradual-process" id="markdown-toc-gradual-process">Gradual Process</a></li> </ul> </li> <li><a href="#layout" id="markdown-toc-layout">Layout</a> <ul> <li><a href="#dirty-bit-system" id="markdown-toc-dirty-bit-system">Dirty Bit System</a></li> <li><a href="#global-and-incremental-layout" id="markdown-toc-global-and-incremental-layout">Global and Incremental Layout</a></li> <li><a href="#asynchronous-and-synchronous-layout" id="markdown-toc-asynchronous-and-synchronous-layout">Asynchronous and Synchronous Layout</a></li> <li><a href="#optimizations" id="markdown-toc-optimizations">Optimizations</a></li> <li><a href="#the-layout-process" id="markdown-toc-the-layout-process">The Layout Process</a></li> <li><a href="#width-calculation" id="markdown-toc-width-calculation">Width Calculation</a></li> <li><a href="#line-breaking" id="markdown-toc-line-breaking">Line Breaking</a></li> </ul> </li> <li><a href="#painting" id="markdown-toc-painting">Painting</a> <ul> <li><a href="#global-and-incremental" id="markdown-toc-global-and-incremental">Global and Incremental</a></li> <li><a href="#the-painting-order" id="markdown-toc-the-painting-order">The Painting Order</a></li> <li><a href="#firefox-display-list" id="markdown-toc-firefox-display-list">Firefox Display List</a></li> <li><a href="#webkit-rectangle-storage" id="markdown-toc-webkit-rectangle-storage">Webkit Rectangle Storage</a></li> </ul> </li> <li><a href="#dynamic-changes" id="markdown-toc-dynamic-changes">Dynamic Changes</a></li> <li><a href="#the-rendering-engines-threads" id="markdown-toc-the-rendering-engines-threads">The Rendering Engine’s Threads</a> <ul> <li><a href="#event-loop" id="markdown-toc-event-loop">Event Loop</a></li> </ul> </li> <li><a href="#css2-visual-model" id="markdown-toc-css2-visual-model">CSS2 Visual Model</a> <ul> <li><a href="#the-canvas" id="markdown-toc-the-canvas">The Canvas</a></li> <li><a href="#css-box-model" id="markdown-toc-css-box-model">CSS Box Model</a></li> <li><a href="#positioning-scheme" id="markdown-toc-positioning-scheme">Positioning Scheme</a></li> <li><a href="#box-types" id="markdown-toc-box-types">Box Types</a></li> <li><a href="#positioning" id="markdown-toc-positioning">Positioning</a> <ul> <li><a href="#relative" id="markdown-toc-relative">Relative</a></li> <li><a href="#floats" id="markdown-toc-floats">Floats</a></li> <li><a href="#absolute-and-fixed" id="markdown-toc-absolute-and-fixed">Absolute and Fixed</a></li> </ul> </li> <li><a href="#layered-representation" id="markdown-toc-layered-representation">Layered Representation</a></li> </ul> </li> <li><a href="#resources" id="markdown-toc-resources">Resources</a></li> </ul> </li> </ul> <h2 id="preface">Preface</h2> <p>This comprehensive primer on the internal operations of WebKit and Gecko is the result of much research done by Israeli developer Tali Garsiel. Over a few years, she reviewed all the published data about browser internals and spent a lot of time reading web browser source code. She wrote:</p> <blockquote> <p>In the years of IE 90% dominance there was nothing much to do but regard the browser as a “black box”, but now, with open source browsers having <a href="http://techcrunch.com/2011/08/01/open-web-browsers/">more than half of the usage share</a>, it’s a good time to take a peek under the engine’s hood and see what’s inside a web browser. Well, what’s inside are millions of C++ lines…</p> </blockquote> <p>Tali published her research on <a href="http://taligarsiel.com/">her site</a>, but we knew it deserved a larger audience, so we’ve cleaned it up and republished it here. As a web developer, <strong>learning the internals of browser operations helps you make better decisions and know the justifications behind development best practices</strong>.</p> <p>You can also watch Tali Garsiel give a talk on this topic on Vimeo:</p> <iframe src="//player.vimeo.com/video/44182484" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe> <p>While this is a rather lengthy document, we recommend you spend some time digging in; we guarantee you’ll be glad you did.</p> <h2 id="introduction">Introduction</h2> <p>Web browsers are probably the most widely used software. In this primer, I will explain how they work behind the scenes. We will see what happens when you type google.com in the address bar until you see the Google page on the browser screen.</p> <h3 id="the-browsers-we-will-talk-about">The Browsers We Will Talk About</h3> <p>There are five major browsers used today - Internet Explorer, Firefox, Safari, Chrome and Opera. I will give examples from the open source browsers - Firefox, Chrome and Safari, which is partly open source. According to the <a href="http://www.w3schools.com/browsers/browsers_stats.asp">W3C browser statistics</a>, currently (September 2011), the usage share of Firefox, Safari, Chrome and Opera is almost 80%. So nowdays open source browsers are a substantial part of the browser business.</p> <h3 id="the-browsers-main-functionality">The Browser’s Main Functionality</h3> <p>The browser main functionality is to present the web resource you choose, by requesting it from the server and displaying it on the browser window. The resource format is usually HTML but can also be PDF, image etc. The location of the resource is specified by the user using a URI (Uniform Resource Identifier). The way the browser interprets and displays HTML files is specified in the HTML and CSS specifications <sup id="fnref:css21"><a href="#fn:css21" class="footnote">1</a></sup>. These specifications are maintained by the <a id="w3c"></a>W3C (World Wide Web Consortium) organization, which is the standards organization for the web.</p> <p>For years browsers conformed to only a part of the specifications and developed their own extensions. That caused serious compatibility issues for web authors. Today most of the browsers more or less conform to the specifications. Browser’s user interfaces have a lot in common with each other. Among the common user interface elements are:</p> <ul> <li>Address bar for inserting the URI</li> <li>Back and forward buttons</li> <li>Bookmarking options</li> <li>Refresh and stop buttons for refreshing and stopping the loading of current documents</li> <li>Home button that gets you to your home page</li> </ul> <p>Strangely enough, the browser’s user interface is not specified in any formal specification, it is just good practices shaped over years of experience and by browsers imitating one another. The HTML5 specification doesn’t define UI elements a browser must have, but lists some common elements. Among those are the address bar, status bar and toolbar. There are, of course, features unique to a specific browser such as Firefox download manager.</p> <h3 id="the-browsers-high-level-structure">The Browser’s High Level Structure</h3> <p>The browser’s main components are <sup id="fnref:refarch"><a href="#fn:refarch" class="footnote">2</a></sup>:</p> <ol> <li>The user interface - this includes the address bar, back/forward button, bookmarking menu etc. Every part of the browser display except the main window where you see the requested page.</li> <li>The browser engine - the interface for querying and manipulating the rendering engine.</li> <li>The rendering engine - responsible for displaying the requested content. For example if the requested content is HTML, it is responsible for parsing the HTML and CSS and displaying the parsed content on the screen.</li> <li>Networking - used for network calls, like HTTP requests <sup id="fnref:lifehttprequest"><a href="#fn:lifehttprequest" class="footnote">3</a></sup> . It has platform independent interface and underneath implementations for each platform.</li> <li>UI backend - used for drawing basic widgets like combo boxes and windows. It exposes a generic interface that is not platform specific. Underneath it uses the operating system user interface methods.</li> <li>JavaScript interpreter. Used to parse and execute the JavaScript code.</li> <li>Data storage. This is a persistence layer. The browser needs to save some data to the hard disk, e.g cookies. The HTML5 specification includes ‘web database’ which is a complete (although light) database in the browser.</li> </ol> <figure> <p><img src="/assets/images/browser-main-components.png" alt="Browser Main Components" /></p> <figcaption>Figure 1: Browser Main Components</figcaption> </figure> <p>It is important to note that Chrome, unlike most browsers, holds multiple instances of the rendering engine - one for each tab. Each tab is a separate process.</p> <h3 id="communication-between-the-components">Communication Between the Components</h3> <p>Both Firefox and Chrome developed a special communication infrastructures.</p> <h2 id="the-rendering-engine">The Rendering Engine</h2> <p>The responsibility of the rendering engine is well… rendering, that is display of the requested contents on the browser screen. By default the rendering engine can display HTML and XML documents and images. It can display other types through plug-ins (browser extensions). An example is displaying a PDF using a PDF viewer plug-in. Here we will focus on the main use case - displaying HTML and images that are formatted using CSS.</p> <h3 id="rendering-engines">Rendering Engines</h3> <p>Our reference browsers - Firefox, Chrome and Safari are built upon two rendering engines. Firefox <sup id="fnref:buildfirefox"><a href="#fn:buildfirefox" class="footnote">4</a></sup> uses Gecko - a “home made” Mozilla rendering engine. Both Safari and Chrome use Webkit. Webkit <sup id="fnref:buildwebkit"><a href="#fn:buildwebkit" class="footnote">5</a></sup> is an open source rendering engine which started as an engine for the Linux platform and was modified by Apple to support Mac and Windows. See <a href="http://webkit.org/">http://webkit.org/</a> for more details.</p> <h3 id="the-main-flow">The Main Flow</h3> <p>The rendering engine will start getting the contents of the requested document from the networking layer. This will usually be done in 8K chunks. After that this is the basic flow of the rendering engine:</p> <figure> <p><img src="/assets/images/rendering-engine-basic-flow.png" alt="Rendering engine basic flow" /></p> <figcaption>Figure 2: Rendering engine basic flow</figcaption> </figure> <p>The rendering engine will start parsing the HTML document and turn the tags to <a href="#DOM">DOM</a> nodes in a tree called the “content tree”.</p> <p>It will parse the style data, both in external CSS files and in style elements. The styling information together with visual instructions in the HTML will be used to create another tree - the <a href="#render-tree-construction">render tree</a>. The render tree contains rectangles with visual attributes like color and dimensions. The rectangles are in the right order to be displayed on the screen. After the construction of the render tree it goes through a “<a href="#layout">layout</a>” process. This means giving each node the exact coordinates where it should appear on the screen.</p> <p>The next stage is <a href="#Painting">painting</a>. The render tree will be traversed and each node will be painted using the UI backend layer. It’s important to understand that this is a gradual process. For better user experience, the rendering engine will try to display contents on the screen as soon as possible. It will not wait until all HTML is parsed before starting to build and layout the render tree. Parts of the content will be parsed and displayed, while the process continues with the rest of the contents that keeps coming from the network.</p> <h3 id="main-flow-examples">Main Flow Examples</h3> <figure> <p><img src="/assets/images/webkit-main-flow.png" alt="Webkit main flow" /></p> <figcaption>Figure 3: Webkit main flow</figcaption> </figure> <figure> <p><img src="/assets/images/mozilla.jpg" alt="Mozilla main flow" /></p> <figcaption>Figure 4: Mozilla's Gecko rendering engine main flow</figcaption> </figure> <p>From figures 3 and 4 you can see that although Webkit and Gecko <sup id="fnref:geckooverview"><a href="#fn:geckooverview" class="footnote">6</a></sup> use slightly different terminology, the flow is basically the same. Gecko calls the tree of visually formatted elements the Frame tree. Each element is a frame. Webkit uses the term “Render Tree” and it consists of “Render Objects”. Webkit uses the term “layout” for the placing of elements, while Gecko calls it “Reflow”. “Attachment” is Webkit’s term for connecting DOM nodes and visual information to create the render tree. A minor non semantic difference is that Gecko has an extra layer between the HTML and the DOM tree. It is called the “content sink” and is a factory for making DOM elements. We will talk about each part of the flow:</p> <h3 id="parsing-and-dom-tree-construction">Parsing and DOM Tree Construction</h3> <h4 id="parsing---general">Parsing - General</h4> <p>Since parsing is a very significant process within the rendering engine, we will go into it a little more deeply. Let’s begin with a little introduction about parsing <sup id="fnref:dragonbook"><a href="#fn:dragonbook" class="footnote">7</a></sup>.</p> <p>Parsing a document means translating it to some structure that makes sense - something the code can understand and use. The result of parsing is usually a tree of nodes that represent the structure of the document. It is called a parse tree or a syntax tree.</p> <p>For example - parsing the expression “2 + 3 - 1” could return this tree:</p> <figure> <p><img src="/assets/images/mathematical-expression-tree-node.png" alt="Mathematical expression node tree" /></p> <figcaption>Figure 5: Mathematical expression node tree</figcaption> </figure> <h5 id="grammars">Grammars</h5> <p>Parsing is based on the syntax rules the document obeys - the language or format it was written in. Every format you can parse must have deterministic grammar consisting of vocabulary and syntax rules. It is called a <a href="#context_free_grammar">context free grammar</a>. Human languages are not such consistent languages and therefore cannot be parsed with conventional parsing techniques.</p> <h5 id="parser---lexer-combination">Parser - Lexer Combination</h5> <p>Parsing can be separated into two sub processes - lexical analysis and syntax analysis. Lexical analysis is the process of breaking the input into tokens. Tokens are the language vocabulary - the collection of valid building blocks. In human language it will consist of all the words that appear in the dictionary for that language. Syntax analysis is the applying of the language syntax rules.</p> <p>Parsers usually divide the work between two components - the <strong>lexer **(sometimes called tokenizer) that is responsible for breaking the input into valid tokens, and the **parser</strong> that is responsible for constructing the parse tree by analyzing the document structure according to the language syntax rules. The lexer knows how to strip irrelevant characters like white spaces and line breaks.</p> <figure> <p><img src="/assets/images/from-source-document-to-parse-trees.png" alt="From source document to parse trees" /></p> <figcaption>Figure 6: From source document to parse trees</figcaption> </figure> <p>The parsing process is iterative. The parser will usually ask the lexer for a new token and try to match the token with one of the syntax rules. If a rule is matched, a node corresponding to the token will be added to the parse tree and the parser will ask for another token. If no rule matches, the parser will store the token internally, and keep asking for tokens until a rule matching all the internally stored tokens is found. If no rule is found then the parser will raise an exception. This means the document was not valid and contained syntax errors.</p> <h5 id="translation">Translation</h5> <p>Many times the parse tree is not the final product. Parsing is often used in translation - transforming the input document to another format. One example is compilation. The compiler that compiles a program into machine code first parses it into a parse tree and then translates the tree into a machine code document.</p> <figure> <p><img src="/assets/images/compilation-flow.png" alt="Compilation flow" /></p> <figcaption>Figure 7: Compilation flow</figcaption> </figure> <h5 id="parsing-example">Parsing Example</h5> <p>In figure 5 we built a parse tree from a mathematical expression. Let’s try to define a simple mathematical language and see the parse process. Vocabulary: Our language can include integers, plus signs and minus signs. Syntax:</p> <ol> <li>The language syntax building blocks are expressions, terms and operations.</li> <li>Our language can include any number of expressions.</li> <li>A expression is defined as a “term” followed by an “operation” followed by another term</li> <li>An operation is a plus token or a minus token</li> <li>A term is an integer token or an expression</li> </ol> <p>Let’s analyze the input “2 + 3 - 1”. The first substring that matches a rule is “2”, according to rule #5 it is a term. The second match is “2 + 3” this matches the second rule - a term followed by an operation followed by another term. The next match will only be hit at the end of the input. “2 + 3 - 1” is an expression because we already know that “2 + 3” is a term so we have a term followed by an operation followed by another term. “2 + +” will not match any rule and therefore is an invalid input.</p> <h5 id="formal-definitions-for-vocabulary-and-syntax">Formal Definitions for Vocabulary and Syntax</h5> <p>Vocabulary is usually expressed by <a href="http://www.regular-expressions.info/">regular expressions</a>. For example our language will be defined as:</p> <figure> <div class="highlight"><pre><code class="language-js"><span class="nx">INTEGER</span><span class="o">:</span> <span class="mi">0</span><span class="o">|</span><span class="p">[</span><span class="mi">1</span><span class="o">-</span><span class="mi">9</span><span class="p">][</span><span class="mi">0</span><span class="o">-</span><span class="mi">9</span><span class="p">]</span><span class="o">*</span> <span class="nx">PLUS</span><span class="o">:</span> <span class="o">+</span> <span class="nx">MINUS</span><span class="o">:</span> <span class="o">-</span></code></pre></div> <figcaption>Example of our language in BNF</figcaption> </figure> <p>As you see, integers are defined by a regular expression. Syntax is usually defined in a format called <a href="http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form">BNF</a>. Our language will be defined as:</p> <figure> <div class="highlight"><pre><code class="language-js"><span class="nx">expression</span> <span class="o">:=</span> <span class="nx">term</span> <span class="nx">operation</span> <span class="nx">term</span> <span class="nx">operation</span> <span class="o">:=</span> <span class="nx">PLUS</span> <span class="o">|</span> <span class="nx">MINUS</span> <span class="nx">term</span> <span class="o">:=</span> <span class="nx">INTEGER</span> <span class="o">|</span> <span class="nx">expression</span></code></pre></div> <figcaption>Language definition in BNF</figcaption> </figure> <p>We said that a language can be parsed by regular parsers if its grammar is a <a id="context_free_grammar"></a>context free grammar. An intuitive definition of a context free grammar is a grammar that can be entirely expressed in BNF. For a formal definition see <a href="http://en.wikipedia.org/wiki/Context-free_grammar">http://en.wikipedia.org/wiki/Context-free_grammar</a></p> <h5 id="types-of-parsers">Types of Parsers</h5> <p>There are two basic types of parsers - top down parsers and bottom up parsers. An intuitive explanation is that top down parsers see the high level structure of the syntax and try to match one of them. Bottom up parsers start with the input and gradually transform it into the syntax rules, starting from the low level rules until high level rules are met. Let’s see how the two types of parsers will parse our example:</p> <p>A top down parser starts from the higher level rule - it identifies “2 + 3” as an expression. It will then identify “2 + 3 - 1” as an expression (the process of identifying the expression evolves matching the other rules, but the start point is the highest level rule).</p> <p>The bottom up parser will scan the input until a rule is matched it will then replace the matching input with the rule. This will go on until the end of the input. The partly matched expression is placed on the parsers stack.</p> <table> <thead> <tr> <th>Stack</th> <th style="text-align: right">Input</th> </tr> </thead> <tbody> <tr> <td> </td> <td style="text-align: right">2 + 3 - 1</td> </tr> <tr> <td>Term</td> <td style="text-align: right">+ 3 - 1</td> </tr> <tr> <td>Term Operation</td> <td style="text-align: right">3 - 1</td> </tr> <tr> <td>Expression</td> <td style="text-align: right">- 1</td> </tr> <tr> <td>Expression operation</td> <td style="text-align: right">1</td> </tr> <tr> <td>Expression</td> <td style="text-align: right"> </td> </tr> </tbody> </table> <p>This type of bottom up parser is called a shift reduce parser, because the input is shifted to the right (imagine a pointer pointing first at the input start and moving to the right) and is gradually reduced to syntax rules.</p> <h5 id="generating-parsers-automatically">Generating Parsers Automatically</h5> <p>There are tools that can generate a parser for you. They are called parser generators. You feed them with the grammar of your language - its vocabulary and syntax rules and they generate a working parser. Creating a parser requires a deep understanding of parsing and its not easy to create an optimized parser by hand, so parser generators can be very useful.</p> <p>Webkit uses two well known parser generators - Flex for creating a lexer and Bison for creating a parser (you might run into them with the names Lex and Yacc). Flex input is a file containing regular expression definitions of the tokens. Bison’s input is the language syntax rules in BNF format.</p> <h4 id="html-parser">HTML Parser</h4> <p>The job of the HTML parser is to parse the HTML markup into a parse tree.</p> <h5 id="the-html-grammar-definition">The HTML Grammar Definition</h5> <p>The vocabulary and syntax of HTML are defined in <a href="#w3c">specifications</a> created by the w3c organization. The current version is HTML4 and work on HTML5 is in progress.</p> <h5 id="not-a-context-free-grammar">Not a Context Free Grammar</h5> <p>As we have seen in the parsing introduction, grammar syntax can be defined formally using formats like BNF. Unfortunately not all of the conventional parser topics apply to HTML (I didn’t bring them up just for fun - they will be used in parsing CSS and JavaScript). HTML cannot easily be defined by a context free grammar that parsers need.</p> <p>There is a formal format for defining HTML - DTD (Document Type Definition) - but it is not a context free grammar. This appears strange at first site - HTML is rather close to XML. There are lots of available XML parsers. There is an XML variation of HTML - XHTML - so what’s the big difference? The difference is that HTML approach is more “forgiving”, it lets you omit certain tags which are added implicitly, sometimes omit the start or end of tags etc.</p> <p>On the whole it’s a “soft” syntax, as opposed to XML’s stiff and demanding syntax. Apparently this seemingly small difference makes a world of a difference. On one hand this is the main reason why HTML is so popular - it forgives your mistakes and makes life easy for the web author. On the other hand, it makes it difficult to write a format grammar. So to summarize - HTML cannot be parsed easily, not by conventional parsers since its grammar is not a context free grammar, and not by XML parsers.</p> <h5 id="html-dtd">HTML DTD</h5> <p>HTML definition is in a DTD format. This format is used to define languages of the <a href="http://en.wikipedia.org/wiki/Standard_Generalized_Markup_Language">SGML</a> family. The format contains definitions for all allowed elements, their attributes and hierarchy. As we saw earlier, the HTML DTD doesn’t form a context free grammar. There are a few variations of the DTD. The strict mode conforms solely to the specifications but other modes contain support for markup used by browsers in the past. The purpose is backwards compatibility with older content. The current strict DTD is at <a href="http://www.w3.org/TR/html4/strict.dtd">http://www.w3.org/TR/html4/strict.dtd</a></p> <h5 id="dom">DOM</h5> <p>The output tree - the parse tree is a tree of DOM element and attribute nodes. DOM is short for Document Object Model. It is the object presentation of the HTML document and the interface of HTML elements to the outside world like JavaScript. The root of the tree is the “<a href="http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001/level-one-core.html#i-Document">document</a>” object.</p> <p>The DOM has an almost one to one relation to the markup. Example, this markup:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span> Hello World <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">div</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;example.png&quot;</span><span class="p">/&gt;&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span></code></pre></div> <figcaption>Example markup</figcaption> </figure> <p>Would be translated to the following DOM tree:</p> <figure> <p><img src="/assets/images/dom-tree-of-the-example-markup.png" alt="DOM tree of the example markup" /></p> <figcaption>Figure 8: DOM tree of the example markup</figcaption> </figure> <p>Like HTML, DOM is specified by the w3c organizationm see <a href="http://www.w3.org/DOM/DOMTR">http://www.w3.org/DOM/DOMTR</a>. It is a generic specification for manipulating documents. A specific module describes HTML specific elements. The HTML definitions can be found here: <a href="http://www.w3.org/TR/2003/REC-DOM-Level-2-HTML-20030109/idl-definitions.html">http://www.w3.org/TR/2003/REC-DOM-Level-2-HTML-20030109/idl-definitions.html</a>. When we say the tree contains DOM nodes, we mean the tree is constructed of elements that implement one of the DOM interfaces. Browsers use concrete implementations that have other attributes used by the browser internally.</p> <h5 id="the-parsing-algorithm">The Parsing Algorithm</h5> <p>As we saw in the previous sections, HTML cannot be parsed using the regular top down or bottom up parsers. The reasons are:</p> <ul> <li>The forgiving nature of the language.</li> <li>The fact that browsers have traditional error tolerance to support well known cases of invalid HTML.</li> <li>The parsing process in reentrant. Usually the source doesn’t change during parsing, but in HTML, script tags containing “document.write” can add extra tokens, so the parsing process actually modifies the input.</li> </ul> <p>Unable to use the regular parsing techniques, browsers create custom parsers for parsing HTML. The parsing algorithm is described in details by the HTML5 specification. The algorithm consists of two stages - tokenization and tree construction. Tokenization is the lexical analysis, parsing the input into tokens. Among HTML tokens are start tags, end tags, attribute names and attribute values. The tokenizer recognizes the token, gives it to the tree constructor and consumes the next character for recognizing the next token and so on until the end of the input.</p> <figure> <p><img src="/assets/images/html-parsing-flow.png" alt="HTML Parsing Flow" /></p> <figcaption>Figure 9: HTML Parsing Flow</figcaption> </figure> <h5 id="the-tokenization-algorithm">The Tokenization Algorithm</h5> <p>The algorithm’s output is an HTML token. The algorithm is expressed as a state machine. Each state consumes one or more characters of the input stream and updates the next state according to those characters. The decision is influenced by the current tokenization state and by the tree construction state. This means the same consumed character will yield different results for the correct next state, depending on the current state. The algorithm is too complex to bring fully, so let’s see a simple example that will help us understand the principal.</p> <p>Basic example - tokenizing the following HTML:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span> Hello world <span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span></code></pre></div> <figcaption>The HTML to be tokenized</figcaption> </figure> <ol> <li>The initial state is the “Data state”. When the <code class="highlight language-javascript"><span class="o">&lt;</span></code> character is encountered, the state is changed to <strong>“Tag open state”</strong>.</li> <li>Consuming an “a-z” character causes creation of a “Start tag token”, the state is changed to <strong>“Tag name state”</strong>.</li> <li>We stay in this state until the <code class="highlight language-javascript"><span class="o">&gt;</span></code> character is consumed. Each character is appended to the new token name. In our case the created token is an “html” token.</li> <li>When the <code class="highlight language-javascript"><span class="o">&gt;</span></code> tag is reached, the current token is emitted and the state changes back to the <strong>“Data state”</strong>.</li> <li>The <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">body</span><span class="o">&gt;</span></code> tag will be treated by the same steps. So far the “html” and “body” tags were emitted. We are now back at the <strong>“Data state”</strong>.</li> <li>Consuming the “H” character of “Hello world” will cause creation and emitting of a character token, this goes on until the <code class="highlight language-javascript"><span class="o">&gt;</span></code> of <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">body</span><span class="o">&gt;</span></code> is reached.</li> <li>We will emit a character token for each character of “Hello world”. We are now back at the <strong>“Tag open state”</strong>.</li> <li>Consuming the next input “/” will cause creation of an “end tag token” and a move to the <strong>“Tag name state”</strong>.</li> <li>Again we stay in this state until we reach <code class="highlight language-javascript"><span class="o">&gt;</span></code>.Then the new tag token will be emitted and we go back to the <strong>“Data state”</strong>.</li> <li>The <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">html</span><span class="o">&gt;</span></code> input will be treated like the previous case.</li> </ol> <figure> <p><img src="/assets/images/tokenizing-the-example-input.png" alt="Tokenizing the example input" /></p> <figcaption>Figure 10: Tokenizing the example input</figcaption> </figure> <h5 id="tree-construction-algorithm">Tree Construction Algorithm</h5> <p>When the parser is created the document object is created. During the tree construction stage the DOM tree with the document in its root will be modified and elements will be added to it. Each node emitted by the tokenizer will be processed by the tree constructor. For each token the specification defines which DOM element is relevant to it and will be created for this token. Except of adding the element to the DOM tree it is also added to a stack of open elements.</p> <p>This stack is used to correct nesting mismatches and unclosed tags. The algorithm is also described as a state machine. The states are called “insertion modes.” Let’s see the tree construction process for the example input:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span> Hello world <span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span></code></pre></div> <figcaption>Example input</figcaption> </figure> <p>The input to the tree construction stage is a sequence of tokens from the tokenization stage:</p> <ol> <li>The first mode is the <strong>“initial mode”</strong>.</li> <li>Receiving the html token will cause a move to the <strong>“before html”</strong> mode and a reprocessing of the token in that mode.</li> <li>This will cause a creation of the HTMLHtmlElement element and it will be appended to the root document object. The state will be changed to <strong>“before head.”</strong></li> <li>We receive the “body” token. An HTMLHeadElement will be created implicitly although we don’t have a “head” token and it will be added to the tree.</li> <li>We now move to the <strong>“in head”</strong> mode and then to <strong>“after head”</strong>.</li> <li>The body token is reprocessed, an HTMLBodyElement is created and inserted and the mode is transferred to <strong>“in body.”</strong></li> <li>The character tokens of the “Hello world” string are now received.</li> <li>The first one will cause creation and insertion of a “Text” node and the other characters will be appended to that node.</li> <li>The receiving of the body end token will cause a transfer to <strong>“after body”</strong> mode.</li> <li>We will now receive the html end tag which will move us to <strong>“after after body”</strong> mode.</li> <li>Receiving the end of file token will end the parsing.</li> </ol> <figure> <p><img src="/assets/images/tree-construction-of-example-html.gif" alt="Tree construction of example HTML" /></p> <figcaption>Figure 11: Tree construction of example HTML</figcaption> </figure> <h5 id="actions-when-the-parsing-is-finished">Actions When the Parsing is Finished</h5> <p>At this stage the browser will mark the document as interactive and start parsing scripts that are in “deferred” mode - those who should be executed after the document is parsed. The document state will then be set to “complete” and a “load” event will be fired. You can see the full algorithms for tokenization and tree construction in HTML5 specification - <a href="http://www.w3.org/TR/html5/syntax.html#html-parser">http://www.w3.org/TR/html5/syntax.html#html-parser</a></p> <h5 id="browsers-error-tolerance">Browsers Error Tolerance</h5> <p>You never get an “Invalid Syntax” error on an HTML page. Browsers fix an invalid content and go on. Take this HTML for example:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">mytag</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">mytag</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">div</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> Really lousy HTML <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span></code></pre></div> <figcaption>Terrible HTML</figcaption> </figure> <p>I must have violated about a million rules (“mytag” is not a standard tag, wrong nesting of the “p” and “div” elements and more) but the browser still shows it correctly and doesn’t complain. So a lot of the parser code is fixing the HTML author’s mistakes. The error handling is quite consistent in browsers but amazingly enough it’s not part of HTML current specification. Like bookmarking and back/forward buttons it’s just something that developed in browsers over the years.</p> <p>There are known invalid HTML constructs that repeat themselves in many sites and the browsers try to fix them in a conformant way with other browsers. The HTML5 specification does define some of these requirements. Webkit summarizes this nicely in the comment at the beginning of the HTML parser class.</p> <blockquote> <p>The parser parses tokenized input into the document, building up the document tree. If the document is well-formed, parsing it is straightforward. Unfortunately, we have to handle many HTML documents that are not well-formed, so the parser has to be tolerant about errors. We have to take care of at least the following error conditions:</p> <ol> <li>The element being added is explicitly forbidden inside some outer tag. In this case we should close all tags up to the one, which forbids the element, and add it afterwards.</li> <li>We are not allowed to add the element directly. It could be that the person writing the document forgot some tag in between (or that the tag in between is optional). This could be the case with the following tags: HTML HEAD BODY TBODY TR TD LI (did I forget any?).</li> <li>We want to add a block element inside to an inline element. Close all inline elements up to the next higher block element.</li> <li>If this doesn’t help, close elements until we are allowed to add the element or ignore the tag.</li> </ol> </blockquote> <p>Let’s see some Webkit error tolerance examples:</p> <h6 id="br-instead-of-br">&lt;/br&gt; Instead of &lt;br&gt;</h6> <p>Some sites use <code class="highlight language-javascript"><span class="o">&lt;</span><span class="err">/br&gt;</span></code> instead of <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">br</span><span class="o">&gt;</span></code>. In order to be compatible with IE and Firefox Webkit treats this like <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">br</span><span class="o">&gt;</span></code>.</p> <p>The code:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">if</span> <span class="p">(</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">isCloseTag</span><span class="p">(</span><span class="n">brTag</span><span class="p">)</span> <span class="o">&amp;&amp;</span> <span class="n">m_document</span><span class="o">-&gt;</span><span class="n">inCompatMode</span><span class="p">())</span> <span class="p">{</span> <span class="n">reportError</span><span class="p">(</span><span class="n">MalformedBRError</span><span class="p">);</span> <span class="n">t</span><span class="o">-&gt;</span><span class="n">beginTag</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>How Webkit treats &lt;/br&gt; as &lt;br&gt;</figcaption> </figure> <p>Note: The error handling is internal - it won’t be presented to the user.</p> <h6 id="a-stray-table">A Stray Table</h6> <p>A stray table is a table inside another tables contents but not inside a table cell.</p> <p>For example:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tr</span><span class="p">&gt;&lt;</span><span class="nt">td</span><span class="p">&gt;</span>inner table<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;&lt;/</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tr</span><span class="p">&gt;&lt;</span><span class="nt">td</span><span class="p">&gt;</span>outer table<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;&lt;/</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">table</span><span class="p">&gt;</span></code></pre></div> <figcaption>HTML resulting in a stray table</figcaption> </figure> <p>Webkit will change the hierarchy to two sibling tables:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tr</span><span class="p">&gt;&lt;</span><span class="nt">td</span><span class="p">&gt;</span>outer table<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;&lt;/</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tr</span><span class="p">&gt;&lt;</span><span class="nt">td</span><span class="p">&gt;</span>inner table<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;&lt;/</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">table</span><span class="p">&gt;</span></code></pre></div> <figcaption>Webkit creates two sibling tables</figcaption> </figure> <p>The code:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">if</span> <span class="p">(</span><span class="n">m_inStrayTableContent</span> <span class="o">&amp;&amp;</span> <span class="n">localName</span> <span class="o">==</span> <span class="n">tableTag</span><span class="p">)</span> <span class="n">popBlock</span><span class="p">(</span><span class="n">tableTag</span><span class="p">);</span></code></pre></div> <figcaption>Webkits code that creates the siblings</figcaption> </figure> <p>Webkit uses a stack for the current element contents - it will pop the inner table out of the outer table stack. The tables will now be siblings.</p> <h6 id="nested-form-elements">Nested Form Elements</h6> <p>In case the user puts a form inside another form, the second form is ignored. The code:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">m_currentFormElement</span><span class="p">)</span> <span class="p">{</span> <span class="n">m_currentFormElement</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HTMLFormElement</span><span class="p">(</span><span class="n">formTag</span><span class="p">,</span> <span class="n">m_document</span><span class="p">);</span> <span class="p">}</span></code></pre></div> <figcaption>Forms inside forms are ignored</figcaption> </figure> <h6 id="a-too-deep-tag-hierarchy">A Too Deep Tag Hierarchy</h6> <p>The quote speaks for itself:</p> <blockquote> <p>www.liceo.edu.mx is an example of a site that achieves a level of nesting of about 1500 tags, all from a bunch of <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">b</span><span class="o">&gt;</span></code> tags. We will only allow at most 20 nested tags of the same type before just ignoring them all together.</p> </blockquote> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="kt">bool</span> <span class="n">HTMLParser</span><span class="o">::</span><span class="n">allowNestedRedundantTag</span><span class="p">(</span><span class="k">const</span> <span class="n">AtomicString</span><span class="o">&amp;</span> <span class="n">tagName</span><span class="p">)</span> <span class="p">{</span> <span class="kt">unsigned</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="k">for</span> <span class="p">(</span><span class="n">HTMLStackElem</span><span class="o">*</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">m_blockStack</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">cMaxRedundantTagDepth</span> <span class="o">&amp;&amp;</span> <span class="n">curr</span> <span class="o">&amp;&amp;</span> <span class="n">curr</span><span class="o">-&gt;</span><span class="n">tagName</span> <span class="o">==</span> <span class="n">tagName</span><span class="p">;</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">-&gt;</span><span class="n">next</span><span class="p">,</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span> <span class="p">}</span> <span class="k">return</span> <span class="n">i</span> <span class="o">!=</span> <span class="n">cMaxRedundantTagDepth</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>Only allow tags nested up to 20 deep</figcaption> </figure> <h6 id="misplaced-html-or-body-end-tags">Misplaced html or body End Tags</h6> <p>Again - the comment speaks for itself. Support for really broken html. We never close the body tag, since some stupid web pages close it before the actual end of the doc. Let’s rely on the end() call to close things:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">if</span> <span class="p">(</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">tagName</span> <span class="o">==</span> <span class="n">htmlTag</span> <span class="o">||</span> <span class="n">t</span><span class="o">-&gt;</span><span class="n">tagName</span> <span class="o">==</span> <span class="n">bodyTag</span> <span class="p">)</span> <span class="k">return</span><span class="p">;</span></code></pre></div> <figcaption>body and HTML tags are only closed on the end call</figcaption> </figure> <p>So web authors beware - unless you want to appear as an example in a Webkit error tolerance code - write well formed HTML.</p> <h4 id="css-parsing">CSS Parsing</h4> <p>Remember the parsing concepts in the introduction <sup id="fnref:stylesystem"><a href="#fn:stylesystem" class="footnote">8</a></sup>? Well, unlike HTML, CSS is a context free grammar and can be parsed using the types of parsers described in the introduction. In fact the CSS specification defines CSS lexical and syntax grammar (<a href="http://www.w3.org/TR/CSS2/grammar.html">http://www.w3.org/TR/CSS2/grammar.html</a>).</p> <p>Let’s see some examples:</p> <p>The lexical grammar (vocabulary) is defined by regular expressions for each token:</p> <figure> <div class="highlight"><pre><code class="language-js"><span class="nx">comment</span> <span class="err">\</span><span class="o">/</span><span class="err">\</span><span class="o">*</span><span class="p">[</span><span class="o">^*</span><span class="p">]</span><span class="o">*</span><span class="err">\</span><span class="o">*+</span><span class="p">([</span><span class="o">^</span><span class="err">/*][^*]*\*+)*\/</span> <span class="nx">num</span> <span class="p">[</span><span class="mi">0</span><span class="o">-</span><span class="mi">9</span><span class="p">]</span><span class="o">+|</span><span class="p">[</span><span class="mi">0</span><span class="o">-</span><span class="mi">9</span><span class="p">]</span><span class="o">*</span><span class="s2">&quot;.&quot;</span><span class="p">[</span><span class="mi">0</span><span class="o">-</span><span class="mi">9</span><span class="p">]</span><span class="o">+</span> <span class="nx">nonascii</span> <span class="p">[</span><span class="err">\</span><span class="mi">200</span><span class="o">-</span><span class="err">\</span><span class="mi">377</span><span class="p">]</span> <span class="nx">nmstart</span> <span class="p">[</span><span class="nx">_a</span><span class="o">-</span><span class="nx">z</span><span class="p">]</span><span class="o">|</span><span class="p">{</span><span class="nx">nonascii</span><span class="p">}</span><span class="o">|</span><span class="p">{</span><span class="nx">escape</span><span class="p">}</span> <span class="nx">nmchar</span> <span class="p">[</span><span class="nx">_a</span><span class="o">-</span><span class="nx">z0</span><span class="o">-</span><span class="mi">9</span><span class="o">-</span><span class="p">]</span><span class="o">|</span><span class="p">{</span><span class="nx">nonascii</span><span class="p">}</span><span class="o">|</span><span class="p">{</span><span class="nx">escape</span><span class="p">}</span> <span class="nx">name</span> <span class="p">{</span><span class="nx">nmchar</span><span class="p">}</span><span class="o">+</span> <span class="nx">ident</span> <span class="p">{</span><span class="nx">nmstart</span><span class="p">}{</span><span class="nx">nmchar</span><span class="p">}</span><span class="o">*</span></code></pre></div> <figcaption>Lexical grammar (vocabulary)</figcaption> </figure> <p>“ident” is short for identifier, like a class name. “name” is an element id (that is referred by “#” )</p> <p>The syntax grammar is described in BNF:</p> <figure> <div class="highlight"><pre><code class="language-js"><span class="nx">ruleset</span> <span class="o">:</span> <span class="nx">selector</span> <span class="p">[</span> <span class="s1">&#39;,&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">selector</span> <span class="p">]</span><span class="o">*</span> <span class="s1">&#39;{&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">declaration</span> <span class="p">[</span> <span class="s1">&#39;;&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">declaration</span> <span class="p">]</span><span class="o">*</span> <span class="s1">&#39;}&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="p">;</span> <span class="nx">selector</span> <span class="o">:</span> <span class="nx">simple_selector</span> <span class="p">[</span> <span class="nx">combinator</span> <span class="nx">selector</span> <span class="o">|</span> <span class="nx">S</span><span class="o">+</span> <span class="p">[</span> <span class="nx">combinator</span><span class="o">?</span> <span class="nx">selector</span> <span class="p">]</span><span class="o">?</span> <span class="p">]</span><span class="o">?</span> <span class="p">;</span> <span class="nx">simple_selector</span> <span class="o">:</span> <span class="nx">element_name</span> <span class="p">[</span> <span class="nx">HASH</span> <span class="o">|</span> <span class="kr">class</span> <span class="o">|</span> <span class="nx">attrib</span> <span class="o">|</span> <span class="nx">pseudo</span> <span class="p">]</span><span class="o">*</span> <span class="o">|</span> <span class="p">[</span> <span class="nx">HASH</span> <span class="o">|</span> <span class="kr">class</span> <span class="o">|</span> <span class="nx">attrib</span> <span class="o">|</span> <span class="nx">pseudo</span> <span class="p">]</span><span class="o">+</span> <span class="p">;</span> <span class="kr">class</span> <span class="o">:</span> <span class="s1">&#39;.&#39;</span> <span class="nx">IDENT</span> <span class="p">;</span> <span class="nx">element_name</span> <span class="o">:</span> <span class="nx">IDENT</span> <span class="o">|</span> <span class="s1">&#39;*&#39;</span> <span class="p">;</span> <span class="nx">attrib</span> <span class="o">:</span> <span class="s1">&#39;[&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">IDENT</span> <span class="nx">S</span><span class="o">*</span> <span class="p">[</span> <span class="p">[</span> <span class="s1">&#39;=&#39;</span> <span class="o">|</span> <span class="nx">INCLUDES</span> <span class="o">|</span> <span class="nx">DASHMATCH</span> <span class="p">]</span> <span class="nx">S</span><span class="o">*</span> <span class="p">[</span> <span class="nx">IDENT</span> <span class="o">|</span> <span class="nx">STRING</span> <span class="p">]</span> <span class="nx">S</span><span class="o">*</span> <span class="p">]</span> <span class="s1">&#39;]&#39;</span> <span class="p">;</span> <span class="nx">pseudo</span> <span class="o">:</span> <span class="s1">&#39;:&#39;</span> <span class="p">[</span> <span class="nx">IDENT</span> <span class="o">|</span> <span class="nx">FUNCTION</span> <span class="nx">S</span><span class="o">*</span> <span class="p">[</span><span class="nx">IDENT</span> <span class="nx">S</span><span class="o">*</span><span class="p">]</span> <span class="s1">&#39;)&#39;</span> <span class="p">]</span> <span class="p">;</span></code></pre></div> <figcaption>Syntax grammar</figcaption> </figure> <p>Explanation: A ruleset is this structure:</p> <figure> <div class="highlight"><pre><code class="language-css"><span class="nt">div</span><span class="p">.</span><span class="nc">error</span><span class="o">,</span> <span class="nt">a</span><span class="p">.</span><span class="nc">error</span> <span class="p">{</span> <span class="k">color</span><span class="p">:</span><span class="kc">red</span><span class="p">;</span> <span class="k">font-weight</span><span class="p">:</span><span class="kc">bold</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>A ruleset</figcaption> </figure> <p>div.error and a.error are selectors. The part inside the curly braces contains the rules that are applied by this ruleset. This structure is defined formally in this definition:</p> <figure> <div class="highlight"><pre><code class="language-js"><span class="nx">ruleset</span> <span class="o">:</span> <span class="nx">selector</span> <span class="p">[</span> <span class="s1">&#39;,&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">selector</span> <span class="p">]</span><span class="o">*</span> <span class="s1">&#39;{&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">declaration</span> <span class="p">[</span> <span class="s1">&#39;;&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="nx">declaration</span> <span class="p">]</span><span class="o">*</span> <span class="s1">&#39;}&#39;</span> <span class="nx">S</span><span class="o">*</span> <span class="p">;</span></code></pre></div> <figcaption>The structure definition</figcaption> </figure> <p>This means a ruleset is a selector or optionally number of selectors separated by a coma and spaces (S stands for white space). A ruleset contains curly braces and inside them a declaration or optionally a number of declarations separated by a semicolon. “declaration” and “selector” will be defined in the following BNF definitions.</p> <h5 id="webkit-css-parser">Webkit CSS Parser</h5> <p>Webkit uses <a href="#generating-parsers-automatically">Flex and Bison</a> parser generators to create parsers automatically from the CSS grammar files. As you may recall from the parser introduction, Bison creates a bottom up shift reduce parser. Firefox uses a top down parser written manually. In both cases each CSS file is parsed into a StyleSheet object, each object contains CSS rules. The CSS rule objects contain selector and declaration objects and other object corresponding to CSS grammar.</p> <figure> <p><img src="/assets/images/parsing-css.png" alt="Parsing CSS" /></p> <figcaption>Figure 12: Parsing CSS</figcaption> </figure> <h4 id="parsing-scripts">Parsing Scripts</h4> <h4 id="the-order-of-processing-scripts-and-stylesheets">The Order of Processing Scripts and Stylesheets</h4> <h5 id="scripts">Scripts</h5> <p>The model of the web is synchronous. Authors expect scripts to be parsed and executed immediately when the parser reaches a <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">script</span><span class="o">&gt;</span></code> tag. The parsing of the document halts until the script is executed. If the script is external then the resource must be first fetched from the network - this is also done synchronously, the parsing halts until the resource is fetched. This was the model for many years and is also specified in HTML 4 <sup id="fnref:html401"><a href="#fn:html401" class="footnote">9</a></sup> and 5 <sup id="fnref:html5"><a href="#fn:html5" class="footnote">10</a></sup> specifications. Authors could mark the script as “defer” and thus it will not halt the document parsing and will execute after it is parsed. HTML5 adds an option to mark the script as asynchronous so it will be parsed and executed by a different thread.</p> <h5 id="speculative-parsing">Speculative Parsing</h5> <p>Both Webkit and Firefox perform this optimization. While executing scripts, another thread parses the rest of the document and finds out which other resources need to be loaded from the network and loads them. This way resources can be loaded on parallel connections and the overall speed is better.</p> <p>Note: the speculative parser doesn’t modify the DOM tree and leaves that to the main parser, it only parses references to external resources like external scripts, style sheets and images.</p> <h5 id="stylesheets">Stylesheets</h5> <p>Stylesheets on the other hand have a different model. Conceptually it seems that since style sheets don’t change the DOM tree, there is no reason to wait for them and stop the document parsing. There is an issue, though, of scripts asking for style information during the document parsing stage. If the style is not loaded and parsed yet, the script will get wrong answers and apparently this caused lots of problems <sup id="fnref:fouc"><a href="#fn:fouc" class="footnote">11</a></sup>. It seems to be an edge case but is actually fairly common. Firefox blocks all scripts when there is a stylesheet that is still being loaded and parsed. Webkit blocks scripts only when they try to access certain style properties that may be effected by unloaded stylesheets.</p> <h3 id="render-tree-construction">Render Tree Construction</h3> <p>While the DOM tree is being constructed, the browser constructs another tree, the render tree. This is a tree of visual elements in the order in which they will be displayed. It is the visual representation of the document. The purpose of this tree is to enable painting the contents in their correct order. Firefox calls the elements in the render tree “frames”. Webkit uses the term renderer or render object. A renderer knows how to layout and paint itself and it’s children. Webkit’s RenderObject class, the base class of the renderers has the following definition:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">class</span> <span class="nc">RenderObject</span> <span class="p">{</span> <span class="k">virtual</span> <span class="kt">void</span> <span class="n">layout</span><span class="p">();</span> <span class="k">virtual</span> <span class="kt">void</span> <span class="nf">paint</span><span class="p">(</span><span class="n">PaintInfo</span><span class="p">);</span> <span class="k">virtual</span> <span class="kt">void</span> <span class="n">rect</span> <span class="nf">repaintRect</span><span class="p">();</span> <span class="n">Node</span><span class="o">*</span> <span class="n">node</span><span class="p">;</span> <span class="c1">//the DOM node</span> <span class="n">RenderStyle</span><span class="o">*</span> <span class="n">style</span><span class="p">;</span> <span class="c1">// the computed style</span> <span class="n">RenderLayer</span><span class="o">*</span> <span class="n">containgLayer</span><span class="p">;</span> <span class="c1">//the containing z-index layer</span> <span class="p">}</span></code></pre></div> <figcaption>Webkit's RenderObject Class</figcaption> </figure> <p>Each renderer represents a rectangular area usually corresponding to the node’s CSS box, as described by the CSS2 spec. It contains geometric information like width, height and position.</p> <p>The box type is affected by the “display” style attribute that is relevant for the node (see the <a href="#style-computation">style computation</a> section). Here is Webkit code for deciding what type of renderer should be created for a DOM node, according to the display attribute.</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="n">RenderObject</span><span class="o">*</span> <span class="n">RenderObject</span><span class="o">::</span><span class="n">createObject</span><span class="p">(</span><span class="n">Node</span><span class="o">*</span> <span class="n">node</span><span class="p">,</span> <span class="n">RenderStyle</span><span class="o">*</span> <span class="n">style</span><span class="p">)</span> <span class="p">{</span> <span class="n">Document</span><span class="o">*</span> <span class="n">doc</span> <span class="o">=</span> <span class="n">node</span><span class="o">-&gt;</span><span class="n">document</span><span class="p">();</span> <span class="n">RenderArena</span><span class="o">*</span> <span class="n">arena</span> <span class="o">=</span> <span class="n">doc</span><span class="o">-&gt;</span><span class="n">renderArena</span><span class="p">();</span> <span class="p">...</span> <span class="n">RenderObject</span><span class="o">*</span> <span class="n">o</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="k">switch</span> <span class="p">(</span><span class="n">style</span><span class="o">-&gt;</span><span class="n">display</span><span class="p">())</span> <span class="p">{</span> <span class="k">case</span> <span class="nl">NONE</span><span class="p">:</span> <span class="k">break</span><span class="p">;</span> <span class="k">case</span> <span class="nl">INLINE</span><span class="p">:</span> <span class="n">o</span> <span class="o">=</span> <span class="k">new</span> <span class="p">(</span><span class="n">arena</span><span class="p">)</span> <span class="n">RenderInline</span><span class="p">(</span><span class="n">node</span><span class="p">);</span> <span class="k">break</span><span class="p">;</span> <span class="k">case</span> <span class="nl">BLOCK</span><span class="p">:</span> <span class="n">o</span> <span class="o">=</span> <span class="k">new</span> <span class="p">(</span><span class="n">arena</span><span class="p">)</span> <span class="n">RenderBlock</span><span class="p">(</span><span class="n">node</span><span class="p">);</span> <span class="k">break</span><span class="p">;</span> <span class="k">case</span> <span class="nl">INLINE_BLOCK</span><span class="p">:</span> <span class="n">o</span> <span class="o">=</span> <span class="k">new</span> <span class="p">(</span><span class="n">arena</span><span class="p">)</span> <span class="n">RenderBlock</span><span class="p">(</span><span class="n">node</span><span class="p">);</span> <span class="k">break</span><span class="p">;</span> <span class="k">case</span> <span class="nl">LIST_ITEM</span><span class="p">:</span> <span class="n">o</span> <span class="o">=</span> <span class="k">new</span> <span class="p">(</span><span class="n">arena</span><span class="p">)</span> <span class="n">RenderListItem</span><span class="p">(</span><span class="n">node</span><span class="p">);</span> <span class="k">break</span><span class="p">;</span> <span class="p">...</span> <span class="p">}</span> <span class="k">return</span> <span class="n">o</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>Webkit's RenderObject::createObject method</figcaption> </figure> <p>The element type is also considered, for example form controls and tables have special frames. In Webkit if an element wants to create a special renderer it will override the <code class="highlight language-javascript"><span class="nx">createRenderer</span><span class="p">()</span></code> method. The renderers points to style objects that contains the non geometric information.</p> <h4 id="the-render-tree-relation-to-the-dom-tree">The Render Tree Relation to the DOM Tree</h4> <p>The renderers correspond to the DOM elements, but the relation is not one to one. Non visual DOM elements will not be inserted in the render tree. An example is the “head” element. Also elements whose display attribute was assigned to “none” will not appear in the tree (elements with “hidden” visibility attribute will appear in the tree) <sup id="fnref:layoutengineinternals"><a href="#fn:layoutengineinternals" class="footnote">12</a></sup> <sup id="fnref:layoutengineinternalsslides"><a href="#fn:layoutengineinternalsslides" class="footnote">13</a></sup>.</p> <p>There are DOM elements which correspond to several visual objects. These are usually elements with complex structures that cannot be described by a single rectangle. For example, the “select” element has 3 renderers - one for the display area, one for the drop down list box and one for the button. Also when text is broken into multiple lines because the width is not sufficient for one line, the new lines will be added as extra renderers. Another example of several renderers is broken HTML.</p> <p>According to CSS spec an inline element must contain either only block elements or only inline elements. In case of mixed content, anonymous block renderers will be created to wrap the inline elements. Some render objects correspond to a DOM node but not in the same place in the tree. Floats and absolutely positioned elements are out of flow, placed in a different place in the tree, and mapped to the real frame. A placeholder frame is where they should have been.</p> <figure> <p><img src="/assets/images/the-render-tree-and-the-corresponding-dom-tree-31.png" alt="The render tree and the corresponding DOM tree" /></p> <figcaption>Figure 13: The render tree and the corresponding DOM tree</figcaption> </figure> <h4 id="the-flow-of-constructing-the-tree">The Flow of Constructing the Tree</h4> <p>In Firefox, the presentation is registered as a listener for DOM updates. The presentation delegates frame creation to the “FrameConstructor” and the constructor resolves style (see <a href="#style-computation">style computation</a>) and creates a frame. In Webkit the process of resolving the style and creating a renderer is called “attachment”. Every DOM node has an “attach” method. Attachment is synchronous, node insertion to the DOM tree calls the new node “attach” method.</p> <p>Processing the html and body tags results in the construction of the render tree root. The root render object corresponds to what the CSS spec calls the containing block - the top most block that contains all other blocks. Its dimensions are the viewport - the browser window display area dimensions. Firefox calls it ViewPortFrame and Webkit calls it RenderView. This is the render object that the document point to. The rest of the tree is constructed as a DOM nodes insertion. See CSS2 on this topic - <a href="http://www.w3.org/TR/CSS21/intro.html#processing-model">http://www.w3.org/TR/CSS21/intro.html#processing-model</a></p> <h4 id="style-computation">Style Computation</h4> <p>Building the render tree requires calculating the visual properties of each render object. This is done by calculating the style properties of each element. The style includes stylesheets of various origins, inline style elements and visual properties in the HTML (like the “bgcolor” property). The latter is translated to matching CSS style properties. The origins of style sheets are the browser’s default style sheets, the style sheets provided by the page author and user style sheets - these are style sheets provides by the browser user (browsers let you define your favorite style. In Firefox, for instance, this is done by placing a style sheet in the “Firefox Profile” folder). Style computation brings up a few difficulties:</p> <ol> <li><a id="issue1"></a>Style data is a very large construct, holding the numerous style properties, this can cause memory problems.</li> <li><a id="issue2"></a>Finding the matching rules for each element can cause performance issues if it’s not optimized. Traversing the entire rule list for each element to find matches is a heavy task. Selectors can have complex structure that can cause the matching process to start on a seemingly promising path that is proven to be futile and another path has to be tried. For example - this compound selector <code class="highlight language-javascript"><span class="nx">div</span> <span class="nx">div</span> <span class="nx">div</span> <span class="nx">div</span> <span class="p">{...</span> <span class="p">}</span></code> means the rules apply to a <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">div</span><span class="o">&gt;</span></code> which is the descendant of 3 divs. Suppose you want to check if the rule applies for a given <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">div</span><span class="o">&gt;</span></code> element. You choose a certain path up the tree for checking. You may need to traverse the node tree up just to find out there are only two divs and the rule does not apply. You then need to try other paths in the tree.</li> <li><a id="issue3"></a>Applying the rules involves quite complex cascade rules that define the hierarchy of the rules.</li> </ol> <p>Let’s see how the browsers face these issues:</p> <h5 id="sharing-style-data">Sharing Style Data</h5> <p>Webkit nodes references style objects (RenderStyle). These objects can be shared by nodes in some conditions. The nodes are siblings or cousins and:</p> <ol> <li>The elements must be in the same mouse state (e.g., one can’t be in :hover while the other isn’t)</li> <li>Neither element should have an id</li> <li>The tag names should match</li> <li>The class attributes should match</li> <li>The set of mapped attributes must be identical</li> <li>The link states must match</li> <li>The focus states must match</li> <li>Neither element should be affected by attribute selectors, where affected is defined as having any selector match that uses an attribute selector in any position within the selector at all</li> <li>There must be no inline style attribute on the elements</li> <li>There must be no sibling selectors in use at all. WebCore <sup id="fnref:webcoreoverview"><a href="#fn:webcoreoverview" class="footnote">14</a></sup> simply throws a global switch when any sibling selector is encountered and disables style sharing for the entire document when they are present. This includes the + selector and selectors like :first-child and :last-child.</li> </ol> <h5 id="firefox-rule-tree">Firefox Rule Tree</h5> <p>Firefox has two extra trees for easier style computation - the rule tree and style context tree <sup id="fnref:stylecontenttree"><a href="#fn:stylecontenttree" class="footnote">15</a></sup>. Webkit also has style objects but they are not stored in a tree like the style context tree, only the DOM node points to its relevant style.</p> <figure> <p><img src="/assets/images/firefox-style-content-tree.png" alt="Firefox Style Content Tree" /></p> <figcaption>Figure 14: Firefox Style Content Tree</figcaption> </figure> <p>The style contexts contain end values. The values are computed by applying all the matching rules in the correct order and performing manipulations that transform them from logical to concrete values. For example - if the logical value is percentage of the screen it will be calculated and transformed to absolute units. The rule tree idea is really clever. It enables sharing these values between nodes to avoid computing them again. This also saves space. All the matched rules are stored in a tree. The bottom nodes in a path have higher priority. The tree contains all the paths for rule matches that were found. Storing the rules is done lazily. The tree isn’t calculated at the beginning for every node, but whenever a node style needs to be computed the computed paths are added to the tree. The idea is to see the tree paths as words in a lexicon. Let’s say we already computed this rule tree:</p> <figure> <p><img src="/assets/images/computed-rule-tree.png" alt="Computed Rule Tree" /></p> <figcaption>Figure 15: Computed Rule Tree</figcaption> </figure> <p>Suppose we need to match rules for another element in the content tree, and find out the matched rules (in the correct order) are B - E - I. We already have this path in the tree because we already computed path A - B - E - I - L. We will now have less work to do. Let’s see how the tree saves as work.</p> <h6 id="division-into-structs">Division Into Structs</h6> <p>The style contexts are divided into structs. Those structs contain style information for a certain categories like border or color. All the properties in a struct are either inherited or non inherited. Inherited properties are properties that unless defined by the element, are inherited from its parent. Non inherited properties (called “reset” properties) use default values if not defined. The tree helps us by caching entire structs (containing the computed end values) in the tree. The idea is that if the bottom node didn’t supply a definition for a struct, a cached struct in an upper node can be used.</p> <h6 id="computing-the-style-contexts-using-the-rule-tree">Computing the Style Contexts Using the Rule Tree</h6> <p>When computing the style context for a certain element, we first compute a path in the rule tree or use an existing one. We then begin to apply the rules in the path to fill the structs in our new style context. We start at the bottom node of the path - the one with the highest precedence (usually the most specific selector) and traverse the tree up until our struct is full. If there is no specification for the struct in that rule node, then we can greatly optimize - we go up the tree until we find a node that specifies it fully and simply point to it - that’s the best optimization - the entire struct is shared. This saves computation of end values and memory.</p> <p>If we find partial definitions we go up the tree until the struct is filled.</p> <p>If we didn’t find any definitions for our struct, then in case the struct is an “inherited” type - we point to the struct of our parent in the <strong>context tree</strong>, in this case we also succeeded in sharing structs. If its a reset struct then default values will be used.</p> <p>If the most specific node does add values then we need to do some extra calculations for transforming it to actual values. We then cache the result in the tree node so it can be used by children.</p> <p>In case an element has a sibling or a brother that points to the same tree node then the <strong>entire style context</strong> can be shared between them.</p> <p>Lets see an example, suppose we have this HTML:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;err&quot;</span> <span class="na">id</span><span class="o">=</span><span class="s">&quot;div1&quot;</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span> this is a <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;big&quot;</span><span class="p">&gt;</span> big error <span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> this is also a <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;big&quot;</span><span class="p">&gt;</span> very big error<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> error <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;err&quot;</span> <span class="na">id</span><span class="o">=</span><span class="s">&quot;div2&quot;</span><span class="p">&gt;</span>another error<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span></code></pre></div> <figcaption>Similar CSS classes share the same style context</figcaption> </figure> <p>And the following rules:</p> <figure> <div class="highlight"><pre><code class="language-css"><span class="nt">div</span> <span class="p">{</span><span class="k">margin</span><span class="p">:</span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="k">color</span><span class="p">:</span><span class="kc">black</span><span class="p">}</span> <span class="p">.</span><span class="nc">err</span> <span class="p">{</span><span class="k">color</span><span class="p">:</span><span class="kc">red</span><span class="p">}</span> <span class="p">.</span><span class="nc">big</span> <span class="p">{</span><span class="k">margin-top</span><span class="p">:</span><span class="mi">3</span><span class="kt">px</span><span class="p">}</span> <span class="nt">div</span> <span class="nt">span</span> <span class="p">{</span><span class="k">margin-bottom</span><span class="p">:</span><span class="mi">4</span><span class="kt">px</span><span class="p">}</span> <span class="p">#</span><span class="nn">div1</span> <span class="p">{</span><span class="k">color</span><span class="p">:</span><span class="kc">blue</span><span class="p">}</span> <span class="p">#</span><span class="nn">div2</span> <span class="p">{</span><span class="k">color</span><span class="p">:</span><span class="kc">green</span><span class="p">}</span></code></pre></div> <figcaption>The CSS</figcaption> </figure> <p>To simplify things let’s say we need to fill out only two structs - the color struct and the margin struct. The color struct contains only one member - the color. The margin struct contains the four sides. The resulting rule tree will look like this (the nodes are marked with the node name : the # of rule they point at):</p> <figure> <p><img src="/assets/images/the-rule-tree.png" alt="The rule tree" /></p> <figcaption>Figure 16: The Rule Tree</figcaption> </figure> <p>The context tree will look like this (node name : rule node they point to):</p> <figure> <p><img src="/assets/images/the-context-tree.jpg" alt="The context tree" /></p> <figcaption>Figure 17: The Context Tree</figcaption> </figure> <p>Suppose we parse the HTML and get to the second <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">div</span><span class="o">&gt;</span></code> tag. We need to create a style context for this node and fill its style structs. We will match the rules and discover that the matching rules for the <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">div</span><span class="o">&gt;</span></code> are 1 ,2 and 6. This means there is already an existing path in the tree that our element can use and we just need to add another node to it for rule 6 (node F in the rule tree). We will create a style context and put it in the context tree. The new style context will point to node F in the rule tree.</p> <p>We now need to fill the style structs. We will begin by filling out the margin struct. Since the last rule node(F) doesn’t add to the margin struct, we can go up the tree until we find a cached struct computed in a previous node insertion and use it. We will find it on node B, which is the uppermost node that specified margin rules. We do have a definition for the color struct, so we can’t use a cached struct. Since color has one attribute we don’t need to go up the tree to fill other attributes. We will compute the end value (convert string to RGB etc) and cache the computed struct on this node.</p> <p>The work on the second <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">span</span><span class="o">&gt;</span></code> element is even easier. We will match the rules and come to the conclusion that it points to rule G, like the previous span. Since we have siblings that point to the same node, we can share the entire style context and just point to the context of the previous span.</p> <p>For structs that contain rules that are inherited from the parent, caching is done on the context tree (the color property is actually inherited, but Firefox treats it as reset and caches it on the rule tree).</p> <p>For instance if we added rules for fonts in a paragraph:</p> <figure> <div class="highlight"><pre><code class="language-css"><span class="nt">p</span> <span class="p">{</span> <span class="k">font-family</span><span class="p">:</span> <span class="n">Verdana</span><span class="p">;</span> <span class="err">font</span> <span class="k">size</span><span class="p">:</span> <span class="mi">10</span><span class="kt">px</span><span class="p">;</span> <span class="k">font-weight</span><span class="p">:</span> <span class="kc">bold</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>Rules for fonts in a paragraph</figcaption> </figure> <p>Then the paragraph element, which is a child of the div in the context tree, could have shared the same font struct as his parent. This is if no font rules where specified for the paragraph.</p> <p>In WebKit, who does not have a rule tree, the matched declarations are traversed four times. First non-important high priority properties are applied (properties that should be applied first because others depend on them, such as display), then high priority important, then normal priority non-important, then normal priority important rules. This means that properties that appear multiple times will be resolved according to the correct cascade order. The last wins.</p> <p>So to summarize: sharing the style objects (entirely or some of the structs inside them) solves issues <a href="#issue1">1</a> and <a href="#issue3">3</a>. The Firefox rule tree also helps in applying the properties in the correct order.</p> <h5 id="manipulating-the-rules-for-an-easy-match">Manipulating the Rules for an Easy Match</h5> <p>There are several sources for style rules:</p> <ul> <li> <p>CSS rules, either in external style sheets or in style elements:<br /> <code class="highlight language-javascript"><span class="nx">p</span> <span class="p">{</span><span class="nx">color</span><span class="o">:</span> <span class="nx">blue</span><span class="p">}</span></code>.</p> </li> <li> <p>Inline style attributes:<br /> <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">p</span> <span class="nx">style</span><span class="o">=</span><span class="s2">&quot;color: blue&quot;</span> <span class="o">/&gt;</span></code></p> </li> <li> <p>HTML visual attributes (which are mapped to relevant style rules):<br /> <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">p</span> <span class="nx">bgcolor</span><span class="o">=</span><span class="s2">&quot;blue&quot;</span> <span class="o">/&gt;</span></code></p> </li> </ul> <p>The last two are easily matched to the element since he owns the style attributes and HTML attributes can be mapped using the element as the key.</p> <p>As noted previously in <a href="#issue2">issue #2</a>, the CSS rule matching can be trickier. To solve the difficulty, the rules are manipulated for easier access.</p> <p>After parsing the stylesheet, the rules are added one of several hash maps, according to the selector. There are maps by id, by class name, by tag name and a general map for anything that doesn’t fit into those categories. If the selector is an id, the rule will be added to the id map, if it’s a class it will be added to the class map etc. This manipulation makes it much easier to match rules. There is no need to look in every declaration - we can extract the relevant rules for an element from the maps. This optimization eliminates 95+% of the rules, so that they need not even be considered during the matching process <sup id="fnref:implementingcss"><a href="#fn:implementingcss" class="footnote">16</a></sup>.</p> <p>Let’s see for example the following style rules:</p> <figure> <div class="highlight"><pre><code class="language-css"><span class="nt">p</span><span class="p">.</span><span class="nc">error</span> <span class="p">{</span><span class="k">color</span><span class="p">:</span> <span class="kc">red</span><span class="p">}</span> <span class="p">#</span><span class="nn">messageDiv</span> <span class="p">{</span><span class="k">height</span><span class="p">:</span> <span class="mi">50</span><span class="kt">px</span><span class="p">}</span> <span class="nt">div</span> <span class="p">{</span><span class="k">margin</span><span class="p">:</span> <span class="mi">5</span><span class="kt">px</span><span class="p">}</span></code></pre></div> <figcaption>These rules will be stored in different maps</figcaption> </figure> <p>The first rule will be inserted into the class map. The second into the id map and the third into the tag map. For the following HTML fragment:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">p</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;error&quot;</span><span class="p">&gt;</span>an error occurred<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">&quot;messageDiv&quot;</span><span class="p">&gt;</span>this is a message<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span></code></pre></div> <figcaption>The accompanying markup</figcaption> </figure> <p>We will first try to find rules for the p element. The class map will contain an “error” key under which the rule for “p.error” is found. The div element will have relevant rules in the id map (the key is the id) and the tag map. So the only work left is finding out which of the rules that were extracted by the keys really match. For example if the rule for the div was:</p> <figure> <div class="highlight"><pre><code class="language-css"><span class="nt">table</span> <span class="nt">div</span> <span class="p">{</span> <span class="k">margin</span><span class="p">:</span> <span class="mi">5</span><span class="kt">px</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>If the rule for the div...</figcaption> </figure> <p>It will still be extracted from the tag map, because the key is the rightmost selector, but it would not match our div element, who does not have a table ancestor. Both Webkit and Firefox do this manipulation.</p> <h5 id="applying-the-rules-in-the-correct-cascade-order">Applying the Rules in the Correct Cascade Order</h5> <p>The style object has properties corresponding to every visual attribute (all css attributes but more generic). If the property is not defined by any of the matched rules - then some properties can be inherited by the parent element style object. Other properties have default values. The problem begins when there is more than one definition - here comes the cascade order to solve the issue.</p> <h6 id="stylesheet-cascade-order">Stylesheet Cascade Order</h6> <p>A declaration for a style property can appear in several style sheets, and several times inside a style sheet. This means the order of applying the rules is very important. This is called the “cascade” order. According to CSS2 spec, the cascade order is (from low to high):</p> <ol> <li>Browser declarations</li> <li>User normal declarations</li> <li>Author normal declarations</li> <li>Author important declarations</li> <li>User important declarations</li> </ol> <p>The browser declarations are least important and the user overrides the author only if the declaration was marked as important. Declarations with the same order will be sorted by <a href="#Specificity">specificity</a> and then the order they are specified. The HTML visual attributes are translated to matching CSS declarations. They are treated as author rules with low priority.</p> <h6 id="specificity">Specificity</h6> <p>The selector specificity is defined by the <a href="http://www.w3.org/TR/CSS2/cascade.html#specificity">CSS2 specification</a> as follows:</p> <ul> <li>count 1 if the declaration is from is a ‘style’ attribute rather than a rule with a selector, 0 otherwise (= a)</li> <li>count the number of ID attributes in the selector (= b)</li> <li>count the number of other attributes and pseudo-classes in the selector (= c)</li> <li>count the number of element names and pseudo-elements in the selector (= d)</li> </ul> <p>Concatenating the four numbers a-b-c-d (in a number system with a large base) gives the specificity. The number base you need to use is defined by the highest count you have in one of the categories. For example, if a=14 you can use hexadecimal base. In the unlikely case where a=17 you will need a 17 digits number base. The later situation can happen with a selector like this: html body div div p… (17 tags in your selector.. not very likely). Some examples:</p> <figure> <div class="highlight"><pre><code class="language-js"> <span class="o">*</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=0 d=0 -&gt; specificity = 0,0,0,0 */</span> <span class="nx">li</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=0 d=1 -&gt; specificity = 0,0,0,1 */</span> <span class="nx">li</span><span class="o">:</span><span class="nx">first</span><span class="o">-</span><span class="nx">line</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=0 d=2 -&gt; specificity = 0,0,0,2 */</span> <span class="nx">ul</span> <span class="nx">li</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=0 d=2 -&gt; specificity = 0,0,0,2 */</span> <span class="nx">ul</span> <span class="nx">ol</span><span class="o">+</span><span class="nx">li</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=0 d=3 -&gt; specificity = 0,0,0,3 */</span> <span class="nx">h1</span> <span class="o">+</span> <span class="o">*</span><span class="p">[</span><span class="nx">rel</span><span class="o">=</span><span class="nx">up</span><span class="p">]{}</span> <span class="cm">/* a=0 b=0 c=1 d=1 -&gt; specificity = 0,0,1,1 */</span> <span class="nx">ul</span> <span class="nx">ol</span> <span class="nx">li</span><span class="p">.</span><span class="nx">red</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=1 d=3 -&gt; specificity = 0,0,1,3 */</span> <span class="nx">li</span><span class="p">.</span><span class="nx">red</span><span class="p">.</span><span class="nx">level</span> <span class="p">{}</span> <span class="cm">/* a=0 b=0 c=2 d=1 -&gt; specificity = 0,0,2,1 */</span> <span class="err">#</span><span class="nx">x34y</span> <span class="p">{}</span> <span class="cm">/* a=0 b=1 c=0 d=0 -&gt; specificity = 0,1,0,0 */</span> <span class="nx">style</span><span class="o">=</span><span class="s2">&quot;&quot;</span> <span class="cm">/* a=1 b=0 c=0 d=0 -&gt; specificity = 1,0,0,0 */</span></code></pre></div> <figcaption>Some examples</figcaption> </figure> <h6 id="sorting-the-rules">Sorting the rules</h6> <p>After the rules are matched, they are sorted according to the cascade rules. Webkit uses bubble sort for small lists and merge sort for big ones. Webkit implements sorting by overriding the <code class="highlight language-javascript"><span class="o">&gt;</span></code> operator for the rules:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">static</span> <span class="kt">bool</span> <span class="k">operator</span> <span class="o">&gt;</span><span class="p">(</span><span class="n">CSSRuleData</span><span class="o">&amp;</span> <span class="n">r1</span><span class="p">,</span> <span class="n">CSSRuleData</span><span class="o">&amp;</span> <span class="n">r2</span><span class="p">)</span> <span class="p">{</span> <span class="kt">int</span> <span class="n">spec1</span> <span class="o">=</span> <span class="n">r1</span><span class="p">.</span><span class="n">selector</span><span class="p">()</span><span class="o">-&gt;</span><span class="n">specificity</span><span class="p">();</span> <span class="kt">int</span> <span class="n">spec2</span> <span class="o">=</span> <span class="n">r2</span><span class="p">.</span><span class="n">selector</span><span class="p">()</span><span class="o">-&gt;</span><span class="n">specificity</span><span class="p">();</span> <span class="k">return</span> <span class="p">(</span><span class="n">spec1</span> <span class="o">==</span> <span class="n">spec2</span><span class="p">)</span> <span class="o">:</span> <span class="n">r1</span><span class="p">.</span><span class="n">position</span><span class="p">()</span> <span class="o">&gt;</span> <span class="n">r2</span><span class="p">.</span><span class="n">position</span><span class="p">()</span> <span class="o">:</span> <span class="n">spec1</span> <span class="o">&gt;</span> <span class="n">spec2</span><span class="p">;</span> <span class="p">}</span></code></pre></div> <figcaption>Webkit overrides the &gt; operator</figcaption> </figure> <h4 id="gradual-process">Gradual Process</h4> <p>Webkit uses a flag that marks if all top level style sheets (including @imports) have been loaded. If the style is not fully loaded when attaching - place holders are used and it s marked in the document, and they will be recalculated once the style sheets are loaded.</p> <h3 id="layout">Layout</h3> <p>When the renderer is created and added to the tree, it does not have a position and size. Calculating these values is called layout or reflow.</p> <p>HTML uses a flow based layout model, meaning that most of the time it is possible to compute the geometry in a single pass. Elements later “in the flow’’ typically do not affect the geometry of elements that are earlier “in the flow’’, so layout can proceed left-to-right, top-to-bottom through the document. There are exceptions - for example, HTML tables may require more than one pass <sup id="fnref:htmlreflow"><a href="#fn:htmlreflow" class="footnote">17</a></sup>.</p> <p>The coordinate system is relative to the root frame. Top and left coordinates are used.</p> <p>Layout is a recursive process. It begins at the root renderer, which corresponds to the <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">html</span><span class="o">&gt;</span></code> element of the HTML document. Layout continues recursively through some or all of the frame hierarchy, computing geometric information for each renderer that requires it.</p> <p>The position of the root renderer is 0, 0 and its dimensions is the viewport - the visible part of the browser window. All renderers have a “layout” or “reflow” method, each renderer invokes the layout method of its children that need layout.</p> <h4 id="dirty-bit-system">Dirty Bit System</h4> <p>In order not to do a full layout for every small change, browser use a “dirty bit” system. A renderer that is changed or added marks itself and its children as “dirty” - needing layout. There are two flags - “dirty” and “children are dirty”. Children are dirty means that although the renderer itself may be ok, it has at least one child that needs a layout.</p> <h4 id="global-and-incremental-layout">Global and Incremental Layout</h4> <p>Layout can be triggered on the entire render tree - this is “global” layout. This can happen as a result of:</p> <ul> <li>A global style change that affects all renderers, like a font size change.</li> <li>As a result of a screen being resized</li> </ul> <p>Layout can be incremental, only the dirty renderers will be layed out (this can cause some damage which will require extra layouts). Incremental layout is triggered (asynchronously) when renderers are dirty. For example when new renderers are appended to the render tree after extra content came from the network and was added to the DOM tree.</p> <figure> <p><img src="/assets/images/incremental-layout.png" alt="Incremental layout" /></p> <figcaption>Figure 18: Incremental layout–only dirty renderers and their children are laid out</figcaption> </figure> <h4 id="asynchronous-and-synchronous-layout">Asynchronous and Synchronous Layout</h4> <p>Incremental layout is done asynchronously. Firefox queues “reflow commands” for incremental layouts and a scheduler triggers batch execution of these commands <sup id="fnref:geckooverview:1"><a href="#fn:geckooverview" class="footnote">6</a></sup>. Webkit also has a timer that executes an incremental layout - the tree is traversed and “dirty” renderers are layout out. Scripts asking for style information, like “offsetHeight” can trigger incremental layout synchronously. Global layout will usually be triggered synchronously. Sometimes layout is triggered as a callback after an initial layout because some attributes, like the scrolling position changed.</p> <h4 id="optimizations">Optimizations</h4> <p>When a layout is triggered by a “resize” or a change in the renderer position(and not size), the renders sizes are taken from a cache and not recalculated. In some cases - only a sub tree is modified and layout does not start from the root. This can happen in cases where the change is local and does not affect its surroundings - like text inserted into text fields (otherwise every keystroke would have triggered a layout starting from the root).</p> <h4 id="the-layout-process">The Layout Process</h4> <p>The layout usually has the following pattern <sup id="fnref:layoutengine"><a href="#fn:layoutengine" class="footnote">18</a></sup>:</p> <ol> <li>Parent renderer determines its own width.</li> <li>Parent goes over children and: <ol> <li>Place the child renderer (sets its x and y).</li> <li>Calls child layout if needed (they are dirty or we are in a global layout or some other reason) - this calculates the child’s height.</li> </ol> </li> <li>Parent uses children accumulative heights and the heights of the margins and paddings to set it own height - this will be used by the parent renderer’s parent.</li> <li>Sets its dirty bit to false.</li> </ol> <p>Firefox uses a “state” object (nsHTMLReflowState) as a parameter to layout (termed “reflow”). Among others the state includes the parents width. The output of Firefox layout is a “metrics” object (nsHTMLReflowMetrics). It will contain the renderer computed height.</p> <h4 id="width-calculation">Width Calculation</h4> <p>The renderer’s width is calculated using the container block’s width , the renderer’s style “width” property, the margins and borders.</p> <p>For example the width of the following div:<br /> <code class="highlight language-javascript"><span class="o">&lt;</span><span class="nx">div</span> <span class="nx">style</span><span class="o">=</span><span class="s2">&quot;width: 30%&quot;</span><span class="o">&gt;&lt;</span><span class="err">/div&gt;</span></code></p> <p>Would be calculated by Webkit as follows (class RenderBox method calcWidth):</p> <ul> <li>The container width is the maximum of the containers availableWidth and 0. The availableWidth in this case is the contentWidth which is calculated as:<br /> <code class="highlight language-javascript"><span class="nx">clientWidth</span><span class="p">()</span> <span class="o">-</span> <span class="nx">paddingLeft</span><span class="p">()</span> <span class="o">-</span> <span class="nx">paddingRight</span><span class="p">()</span></code> clientWidth and clientHeight represent the interior of an object excluding border and scrollbar.</li> <li>The elements width is the “width” style attribute. It will be calculated as an absolute value by computing the percentage of the container width.</li> <li>The horizontal borders and paddings are now added.</li> </ul> <p>So far this was the calculation of the “preferred width.” Now the minimum and maximum widths will be calculated. If the preferred width is higher then the maximum width - the maximum width is used. If it is lower then the minimum width (the smallest unbreakable unit) then the minimum width is used. The values are cached, in case a layout is needed but the width does not change.</p> <h4 id="line-breaking">Line Breaking</h4> <p>When a renderer in the middle of layout decides it needs to break. It stops and propagates to its parent it needs to be broken. The parent will create the extra renderers and calls layout on them.</p> <h3 id="painting">Painting</h3> <p>In the painting stage, the render tree is traversed and the renderers “paint” method is called to display their content on the screen. Painting uses the UI infrastructure component.</p> <h4 id="global-and-incremental">Global and Incremental</h4> <p>Like layout, painting can also be global - the entire tree is painted - or incremental. In incremental painting, some of the renderers change in a way that does not affect the entire tree. The changed renderer invalidates it’s rectangle on the screen. This causes the OS to see it as a “dirty region” and generate a “paint” event. The OS does it cleverly and coalesces several regions into one.</p> <p>In Chrome it is more complicated because the renderer is in a different process then the main process. Chrome simulates the OS behavior to some extent. The presentation listens to these events and delegates the message to the render root. The tree is traversed until the relevant renderer is reached. It will repaint itself (and usually its children).</p> <h4 id="the-painting-order">The Painting Order</h4> <p>CSS2 defines the order of the painting process - <a href="http://www.w3.org/TR/CSS21/zindex.html">http://www.w3.org/TR/CSS21/zindex.html</a>. This is actually the order in which the elements are stacked in the <a href="#stacking_contexts">stacking contexts</a>. This order affects painting since the stacks are painted from back to front. The stacking order of a block renderer is:</p> <ol> <li>background color</li> <li>background image</li> <li>border</li> <li>children</li> <li>outline</li> </ol> <h4 id="firefox-display-list">Firefox Display List</h4> <p>Firefox goes over the render tree and builds a display list for the painted rectangle. It contains the renderers relevant for the rectangle, in the right painting order (backgrounds of the renderers, then borders etc). That way the tree needs to be traversed only once for a repaint instead of several times - painting all backgrounds, then all images , then all borders etc. Firefox optimizes the process by not adding elements that will be hidden, like elements completely beneath other opaque elements.</p> <h4 id="webkit-rectangle-storage">Webkit Rectangle Storage</h4> <p>Before repainting, webkit saves the old rectangle as a bitmap. It then paints only the delta between the new and old rectangles.</p> <h3 id="dynamic-changes">Dynamic Changes</h3> <p>The browsers try to do the minimal possible actions in response to a change. So changes to an elements color will cause only repaint of the element. Changes to the element position will cause layout and repaint of the element, its children and possibly siblings. Adding a DOM node will cause layout and repaint of the node. Major changes, like increasing font size of the “html” element, will cause invalidation of caches and repaint of the entire tree.</p> <h3 id="the-rendering-engines-threads">The Rendering Engine’s Threads</h3> <p>The rendering engine is single threaded. Almost everything, except network operations, happens in a single thread. In Firefox and safari this is the main thread of the browser. In Chrome it’s the tab process main thread <sup id="fnref:webcorerendering"><a href="#fn:webcorerendering" class="footnote">19</a></sup>. Network operations can be performed by several parallel threads. The number of parallel connections is limited (usually 2 - 6 connections. Firefox 3, for example, uses 6).</p> <h4 id="event-loop">Event Loop</h4> <p>The browser main thread is an event loop. It’s an infinite loop that keeps the process alive. It waits for events (like layout and paint events) and processes them. This is Firefox code for the main event loop:</p> <figure> <div class="highlight"><pre><code class="language-cpp"><span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="n">mExiting</span><span class="p">)</span> <span class="n">NS_ProcessNextEvent</span><span class="p">(</span><span class="kr">thread</span><span class="p">);</span></code></pre></div> <figcaption>Firefox event loop</figcaption> </figure> <h3 id="css2-visual-model">CSS2 Visual Model</h3> <h4 id="the-canvas">The Canvas</h4> <p>According to <a href="http://www.w3.org/TR/CSS21/intro.html#processing-model">CCS2 specification</a>, the term canvas describes “the space where the formatting structure is rendered.” - where the browser paints the content. The canvas is infinite for each dimension of the space but browsers choose an initial width based on the dimensions of the viewport. According to <a href="http://www.w3.org/TR/CSS2/zindex.html">http://www.w3.org/TR/CSS2/zindex.html</a>, the canvas is transparent if contained within another, and given a browser defined color if it is not.</p> <h4 id="css-box-model">CSS Box Model</h4> <p>The <a href="http://www.w3.org/TR/CSS2/box.html">CSS box model</a> describes the rectangular boxes that are generated for elements in the document tree and laid out according to the visual formatting model. Each box has a content area (e.g., text, an image, etc.) and optional surrounding padding, border, and margin areas.</p> <figure> <p><img src="/assets/images/css2-box-model.jpg" alt="CSS2 box model" /></p> <figcaption>Figure 19: CSS2 Box Model</figcaption> </figure> <p>Each node generates 0..n such boxes. All elements have a “display” property that determines their type of box that will be generated. Examples:</p> <figure> <div class="highlight"><pre><code class="language-js"><span class="nx">block</span><span class="o">:</span> <span class="nx">generates</span> <span class="nx">a</span> <span class="nx">block</span> <span class="nx">box</span><span class="p">.</span> <span class="nx">inline</span><span class="o">:</span> <span class="nx">generates</span> <span class="nx">one</span> <span class="nx">or</span> <span class="nx">more</span> <span class="nx">inline</span> <span class="nx">boxes</span><span class="p">.</span> <span class="nx">none</span><span class="o">:</span> <span class="nx">no</span> <span class="nx">box</span> <span class="nx">is</span> <span class="nx">generated</span><span class="p">.</span></code></pre></div> <figcaption>Display property possible values</figcaption> </figure> <p>The default is inline but the browser style sheet set other defaults. For example - the default display for “div” element is block. You can find a default style sheet example here <a href="http://www.w3.org/TR/CSS2/sample.html">http://www.w3.org/TR/CSS2/sample.html</a></p> <h4 id="positioning-scheme">Positioning Scheme</h4> <p>There are three schemes:</p> <ul> <li>Normal - the object is positioned according to its place in the document - this means its place in the render tree is like its place in the dom tree and layed out according to its box type and dimensions</li> <li>Float - the object is first layed out like normal flow, then moved as far left or right as possible</li> <li>Absolute - the object is put in the render tree differently than its place in the DOM tree</li> </ul> <p>The positioning scheme is set by the “position” property and the “float” attribute.</p> <ul> <li>static and relative cause a normal flow</li> <li>absolute and fixed cause an absolute positioning</li> </ul> <p>In static positioning no position is defined and the default positioning is used. In the other schemes, the author specifies the position - top, bottom, left, right. The way the box is layed out is determined by:</p> <ul> <li>Box type</li> <li>Box dimensions</li> <li>Positioning scheme</li> <li>External information - like images size and the size of the screen</li> </ul> <h4 id="box-types">Box Types</h4> <p>Block box: forms a block - have their own rectangle on the browser window.</p> <figure> <p><img src="/assets/images/block-box.png" alt="Block box" /></p> <figcaption>Figure 20: Block Box</figcaption> </figure> <p>Inline box: does not have its own block, but is inside a containing block.</p> <figure> <p><img src="/assets/images/inline-boxes.png" alt="Inline boxes" /></p> <figcaption>Figure 21: Inline Boxes</figcaption> </figure> <p>Blocks are formatted vertically one after the other. Inlines are formatted horizontally.</p> <figure> <p><img src="/assets/images/block-and-inline-formatting.png" alt="Block and Inline formatting" /></p> <figcaption>Figure 22: Block and Inline formatting</figcaption> </figure> <p>Inline boxes are put inside lines or “line boxes.” The lines are at least as tall as the tallest box but can be taller, when the boxes are aligned “baseline” - meaning the bottom part of an element is aligned at a point of another box other then the bottom. In case the container width is not enough, the inlines will be put in several lines. This is usually what happens in a paragraph.</p> <figure> <p><img src="/assets/images/lines.png" alt="Lines" /></p> <figcaption>Figure 23: Lines</figcaption> </figure> <h4 id="positioning">Positioning</h4> <h5 id="relative">Relative</h5> <p>Relative positioning - positioned like usual and then moved by the required delta.</p> <figure> <p><img src="/assets/images/relative-positioning.png" alt="Relative positioning" /></p> <figcaption>Figure 24: Relative positioning</figcaption> </figure> <h5 id="floats">Floats</h5> <p>A float box is shifted to the left or right of a line. The interesting feature is that the other boxes flow around it The HTML:</p> <figure> <div class="highlight"><pre><code class="language-html"><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">img</span> <span class="na">style</span><span class="o">=</span><span class="s">&quot;float: right&quot;</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;images/image.gif&quot;</span> <span class="na">width</span><span class="o">=</span><span class="s">&quot;100&quot;</span> <span class="na">height</span><span class="o">=</span><span class="s">&quot;100&quot;</span><span class="p">&gt;</span> Lorem ipsum dolor sit amet, consectetuer... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span></code></pre></div> <figcaption>An image floated to the right</figcaption> </figure> <p>Will look like:</p> <figure> <p><img src="/assets/images/float.png" alt="Float" /></p> <figcaption>Figure 25: Float</figcaption> </figure> <h5 id="absolute-and-fixed">Absolute and Fixed</h5> <p>The layout is defined exactly regardless of the normal flow. The element does not participate in the normal flow. The dimensions are relative to the container. In fixed - the container is the view port.</p> <figure> <p><img src="/assets/images/fixed-positioning.png" alt="Fixed positioning" /></p> <figcaption>Figure 26: Fixed positioning</figcaption> </figure> <p>Note - the fixed box will not move even when the document is scrolled!</p> <h4 id="layered-representation">Layered Representation</h4> <p>It is specified by the z-index CSS property. It represents the 3rd dimension of the box, its position along the “z axis”. The boxes are divided into stacks (called <a id="stacking_contexts">stacking contexts</a>). In each stack the back elements will be painted first and the forward elements on top, closer to the user. In case of overlap it will hide the former element. The stacks are ordered according to the z-index property. Boxes with “z-index” property form a local stack. The viewport has the outer stack. Example:</p> <figure> <div class="highlight"><pre><code class="language-css"><span class="o">&lt;</span><span class="nt">style</span> <span class="nt">type</span><span class="o">=</span><span class="s2">&quot;text/css&quot;</span><span class="o">&gt;</span> <span class="nt">div</span> <span class="p">{</span> <span class="k">position</span><span class="p">:</span> <span class="kc">absolute</span><span class="p">;</span> <span class="k">left</span><span class="p">:</span> <span class="mi">2</span><span class="kt">in</span><span class="p">;</span> <span class="k">top</span><span class="p">:</span> <span class="mi">2</span><span class="kt">in</span><span class="p">;</span> <span class="p">}</span> <span class="o">&lt;/</span><span class="nt">style</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="nt">p</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="nt">div</span> <span class="nt">style</span><span class="o">=</span><span class="s2">&quot;z-index: 3;background-color:red; width: 1in; height: 1in;&quot;</span><span class="o">&gt;&lt;/</span><span class="nt">div</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="nt">div</span> <span class="nt">style</span><span class="o">=</span><span class="s2">&quot;z-index: 1;background-color:green;width: 2in; height: 2in;&quot;</span><span class="o">&gt;&lt;/</span><span class="nt">div</span><span class="o">&gt;</span> <span class="o">&lt;/</span><span class="nt">p</span><span class="o">&gt;</span></code></pre></div> <figcaption>Stacking nodes</figcaption> </figure> <p>The result will be this:</p> <figure> <p><img src="/assets/images/position-fixed.png" alt="Fixed positioning" /></p> <figcaption>Figure 27: Fixed positioning</figcaption> </figure> <p>Although the green div comes before the red one, and would have been painted before in the regular flow, the z-index property is higher, so it is more forward in the stack held by the root box.</p> <h3 id="resources">Resources</h3> <div class="footnotes"> <ol> <li id="fn:css21"> <p><a href="http://www.w3.org/TR/CSS2/">Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification.</a> <a href="#fnref:css21" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:refarch"> <p>Grosskurth, Alan. <a href="http://grosskurth.ca/papers/browser-refarch.pdf">A Reference Architecture for Web Browsers.</a> <a href="#fnref:refarch" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:lifehttprequest"> <p>Alexander Larsson, <a href="https://developer.mozilla.org/en/The_life_of_an_HTML_HTTP_request">The life of an HTML HTTP request.</a> <a href="#fnref:lifehttprequest" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:buildfirefox"> <p><a href="https://developer.mozilla.org/Build_Documentation">How to build Firefox.</a> <a href="#fnref:buildfirefox" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:buildwebkit"> <p><a href="http://webkit.org/building/build.html">How to build WebKit.</a> <a href="#fnref:buildwebkit" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:geckooverview"> <p>Chris Waterson, <a href="http://www.mozilla.org/newlayout/doc/gecko-overview.htm">Gecko Overview.</a> <a href="#fnref:geckooverview" class="reversefootnote">&#8617;</a> <a href="#fnref:geckooverview:1" class="reversefootnote">&#8617;<sup>2</sup></a></p> </li> <li id="fn:dragonbook"> <p>Aho, Sethi, Ullman, <a href="https://www.amazon.com/Compilers-Principles-Techniques-Tools-2nd/dp/0321486811/ref=sr_1_1?s=books&amp;ie=UTF8&amp;qid=1476980885&amp;sr=1-1">Compilers: Principles, Techniques, and Tools (aka the “Dragon book”), Addison-Wesley, 1986</a> <a href="#fnref:dragonbook" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:stylesystem"> <p>L. David Baron, <a href="http://www.mozilla.org/newlayout/doc/style-system.html">Mozilla Style System Documentation.</a> <a href="#fnref:stylesystem" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:html401"> <p><a href="http://www.w3.org/TR/html4/">HTML 4.01 Specification.</a> <a href="#fnref:html401" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:html5"> <p><a href="http://dev.w3.org/html5/spec/Overview.html">W3C HTML5 Specification.</a> <a href="#fnref:html5" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:fouc"> <p>David Hyatt, <a href="https://webkit.org/blog/66/the-fouc-problem/">The FOUC Problem</a> <a href="#fnref:fouc" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:layoutengineinternals"> <p>L. David Baron, <a href="https://www.youtube.com/watch?v=a2_6bGNZ7bA">Faster HTML and CSS: Layout Engine Internals for Web Developers (Google tech talk video).</a> <a href="#fnref:layoutengineinternals" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:layoutengineinternalsslides"> <p>L. David Baron, <a href="http://dbaron.org/talks/2008-11-12-faster-html-and-css/slide-6.xhtml">Faster HTML and CSS: Layout Engine Internals for Web Developers.</a> <a href="#fnref:layoutengineinternalsslides" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:webcoreoverview"> <p>David Hyatt, <a href="http://weblogs.mozillazine.org/hyatt/WebCore/chapter2.html">An Overview of WebCore</a> <a href="#fnref:webcoreoverview" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:stylecontenttree"> <p>Rick Jelliffe. <a href="http://broadcast.oreilly.com/2009/05/the-bold-and-the-beautiful-two.html">The Bold and the Beautiful: two new drafts ML 5.</a> <a href="#fnref:stylecontenttree" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:implementingcss"> <p>David Hyatt, <a href="http://weblogs.mozillazine.org/hyatt/archives/cat_safari.html">Implementing CSS (part 1)</a> <a href="#fnref:implementingcss" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:htmlreflow"> <p>Chris Waterson, <a href="http://www.mozilla.org/newlayout/doc/reflow.html">Notes on HTML Reflow.</a> <a href="#fnref:htmlreflow" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:layoutengine"> <p>L. David Baron, <a href="http://www.mozilla.org/newlayout/doc/layout-2006-07-12/slide-6.xhtml">Mozilla’s Layout Engine.</a> <a href="#fnref:layoutengine" class="reversefootnote">&#8617;</a></p> </li> <li id="fn:webcorerendering"> <p>David Hyatt, <a href="http://webkit.org/blog/114/">WebCore Rendering</a> <a href="#fnref:webcorerendering" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> Sat, 29 Oct 2011 16:53:00 +0100 http://flailingmonkey.com/how-browsers-work/ http://flailingmonkey.com/how-browsers-work/ New job at Mozilla Mike Ratcliffe <p>Okay, I have had the job now for over 3 months but I have really been neglecting my blog… after all, you all deserve to know about what we Mozillians are up to and we are doing some awesome things at the moment.</p> <p>I have been contributing to Firebug and Firebug Lite under various guises now since about 2006 and, I may be biased but, in my opinion it is still the best web development tool ever created. I contribute mainly to the highlighter that outlines nodes on a page but all contributers fix code pretty much anywhere in the source. I remember seeing some interesting code early in 1998 concerning a special built in page triggered by typing <a href="about:mozilla">about:mozilla</a> in the address bar… this is around the time that I learned about open source software and decided that I wanted to work for Mozilla.</p> <p>A couple of years after that, I started contributing to OSCommerce and around 2006 I started contributing to Joe Hewitt’s <a href="http://www.getfirebug.com">Firebug</a>. I have seen Firebug grow from being a simple logging tool to being the fantastic web development tool that it is today. In the past couple of years <a href="http://antennasoft.net/robcee/">Rob Campbell</a> contacted me a few times and told me that I should apply for a position with Mozilla. At the time I thought that all of the Firefox source code was heavy duty C++ code and that makes my eyes bleed, probably something to do with me not coming equipped with a Unix beard.</p> <figure> <p><img src="/assets/images/style-inspector.jpg" alt="Style Inspector" /></p> <figcaption>Style Inspector</figcaption> </figure> <p>Rob invited me again in January 2011 and I had a strong gut feeling that I should go for it no matter what, I could not let this fantastic opportunity pass me by after waiting for almost 14 years. I waited 4 months for Mozilla to contact me and turned down numerous permanent positions with other companies. The wait was worth it even though this position was only a 3 month contract. I would be a member of the Firefox team working on a new set of builtin <a href="http://wiki.mozilla.org/DevTools/Roadmap">Developer Tools</a>. The range of tools that we are building really is amazing, you will hear about them in future blog posts.</p> <p>The developer tool that I have been assigned to work on is called the <a href="http://wiki.mozilla.org/DevTools/Features/StyleInspector">style inspector</a>. It basically displays a node’s calculated styles along with displaying the CSS selectors that are responsible for that style. It also displays a list of selectors that match or do not match your node. I will blog about the tools under development, including my own so that you can have an idea what is in the pipeline.</p> <p>I should point out that the DevTools project does not aim to replace Firebug. Firebug is a fantastic tool that has taken years to develop and will have a bright and shiny future… our DevTools will simply add more functionality.</p> <p>We all know that to build the world’s best developer toolset we need feedback from the whole range of web developers and designers. There are thousands of people around the world that are cursing browser vendors because they have not created a tool to do a particular thing. If this is you then post a comment and let us know, we can’t do anything without you asking until we create the mind reading DevTool and that may be a long time in coming ;o)</p> <p>Does this news excite you? Have an idea for a new developer tool? Let us know by adding a comment.</p> Tue, 26 Jul 2011 22:01:00 +0100 http://flailingmonkey.com/new-job-at-mozilla/ http://flailingmonkey.com/new-job-at-mozilla/ Firefox 4, not just another version number! Mike Ratcliffe <p>So, it has been a long time coming, but I am sure that once you have experienced the awesome that the Mozilla developers have produced you will agree…. this is a download worth getting!</p> <p>We have come a long way since I started using Mozilla products. Way back in 2005 when I was a younger boy (what… I’m only 18. I downloaded and rediscovered the web. What did I download? Why I downloaded <a href="http://www.mozilla.com/en-US/firefox/releases/1.5.html">Firefox 1.5</a>. What a difference and turn that put in the path that is my life.</p> <p>Since then I have seen the release of the wonderful <a href="http://www.mozilla.com/firefox/2.0/releasenotes/">Firefox 2.0</a>, <a href="http://www.mozilla.com/firefox/3.0/releasenotes/">Firefox 3.0</a>, <a href="http://www.mozilla.com/firefox/3.5/releasenotes/">Firefox 3.5</a>, and until recently <a href="http://www.mozilla.com/firefox/3.6/releasenotes/">Firefox 3.6</a>. With each new iteration there have been new features, new websites taking advantage of the packed in technologies, and more importantly a better, safer, and more open web.</p> <iframe src="http://www.youtube.com/embed/tP_i5j74rXo?fs=1" allowfullscreen=""></iframe> <p>So why the wait for <a href="http://www.mozilla.org/firefox/4.0/releasenotes/">Firefox 4</a>? Does the web not need the love and care it once got from Mozilla? I think that if you respond to that one with a yes then you are misinformed. The wait has not been through lack of caring, quite the opposite! There are now so many new features and capabilities that have been added since 3.6 that it just took a little longer to package them all up for you.</p> <p>There is more it it than just the new features though… since the beginning Mozilla Firefox has helped grow a global community of dedicated volunteers who are all working towards a common goal. That goal is a better web… it is a never ending mission and as such if you should ever meet one of these people, even if you are one yourself, don’t forget to thank them for their time, effort and dedication, because with out them…. well I hate to think what the web would be like.</p> <p>Anyhow, that is my boring ramblings… want a full list of features? Visit <a href="http://www.mozilla.org/firefox">www.mozilla.org/firefox</a></p> <p>See <a href="https://developer.mozilla.org/en-US/docs/Web/Demos_of_open_web_technologies">some of the awesome new technology</a> packed into the desktop release:</p> <iframe src="http://www.youtube.com/embed/fdM1SZidPvE?fs=1" allowfullscreen=""></iframe> <p>Oh and don’t forget… Firefox is not only for the desktop! Take the web with you with Firefox 4 Mobile!</p> <iframe src="http://www.youtube.com/embed/5LewqstgWaE?fs=1" allowfullscreen=""></iframe> Wed, 23 Mar 2011 22:47:00 +0000 http://flailingmonkey.com/firefox-4-not-just-another-version-number/ http://flailingmonkey.com/firefox-4-not-just-another-version-number/ Firebug Inspector Improvements Mike Ratcliffe <p>Almost a year since I last posted, I can’t believe it. I am no longer spending time working on Firebug Lite, but am spending my spare time these days working on Firebug’s inspector.</p> <p>There are three versions of the inspector’s highlighter:</p> <ol> <li>The frame highlighter is the blue frame that appears around elements when you click the inspect button and move the mouse cursor around the screen.</li> <li>The box highlighter appears when you mouse over HTML nodes in the HTML panel… this highlighter covers the element with a blue box, the element’s margin with a yellow box etc. this is similar to the inspector used in the layout side panel except that the layout panel version can also display rulers in the elements parent.</li> <li>The image map highlighter appears when you inspect an image that has a related image map.</li> </ol> <p>There have been a whole host of bugs that we have worked on when it comes to the inspector. These bugs can be neatly divided into 4:</p> <ol> <li>Inspector not playing nicely with framesets… because calculating the position of elements within framesets is much more difficult there are naturally a lot of offset problems.</li> <li>People sometimes do some whacky things with web pages like setting the html tag to position absolute etc. This can affects the position of every element on a page, so we needed to make changes to get this working properly.</li> <li>Disabled elements could not be inspected. Because inspected elements block all events, not only click events, no mouseover events are triggered, because the inspector needs this event to inspect an element this was not possible.</li> <li>Because the inspector consists of a set of divs injected into a page the CSS styles from the page were often inherited by the inspector’s divs.</li> </ol> <p>Finding solutions for these problems took time and sanity. The solution to points 1 &amp; 2 were simple enough… simply calculate offsets and apply them to the highlighter. The fix for point 3 was to add proxy elements to the page to enable disabled elements to be inspected, these proxies are transparent divs that are positioned directly over disabled elements. Inspecting these proxy divs simply passed the disabled element to the inspector so that it could be highlighted. The solution to point 4 was to add styles to the highlighters divs with !important to set margins, padding etc. to zero to cancel out any CSS that the inspector would inherit.</p> <p>Since fixing these problems there have only been small issues with the inspector, so there has really not been very much for me to do but one thing has become very apparent… these fixes are not good enough! Apart from the fact that we should never be injecting elements into web pages there are many shortcomings to these fixes. The proxy elements are injected into a page when you click inspect, if a disabled element is moved after you begin inspecting then the proxy and highlighter will be in the wrong position. We could change this to monitor the disabled elements and update the proxies as appropriate but that would cause big performance issues on lots of pages. The biggest problem is with fix number 4…</p> <p>By choosing CSS styles and setting them to 0 !important we effectively stop the elements from inheriting css styles, but the list of elements that we need to do this with is becoming much larger. One of the first styles that we blocked was “-moz-opacity” but then Firefox chose to support “opacity”. Then CSS transforms came along and whichever new css attributes come along all of the time. This is what Ryan Wolf discusses <a href="http://web.archive.org/web/20111001031820/http://borderstylo.com/posts/177-adding-nodes-to-the-dom-with-style">here</a>… he is currently working on the Glass extension (now defunkt) and has had a similar set of experiences in making that extension inspector work.</p> <p>I have known for a long time that we would ultimately need to stop injecting the highlighters into the target page and move them into the browsers chrome. It made sense to use a transparent panel and draw shapes on the panel as somebody inspects the page. To do this I created a prototype that used a canvas in a transparent panel to render the shapes and this appeared to work well minus a couple of performance problems. I hit another brick wall with canvas… it is not possible to use transforms on canvas. I had a discussion with a Firefox developer recently who said that it would make sense to use SVG instead because the canvas was just used for displaying geometric shapes. I have since discovered that SVG will also allow transforms to be applied to the shapes that are drawn on it, this is the direction that I am planning on taking the inspector. Here is how I would like it to work:</p> <p>When a user clicks on inspect the web page is overlayed by a transparent panel that contains SVG content. This overlay will listen for mousemove events and use getElementFromPoint to get the element to be inspected. There will be three methods that the inspector calls and these will highlight the element using the SVG versions of the frame highlighter, the box highlighter or the image map highlighter. Using this transparent overlay has some great advantages that I had not really considered very much. One thing that people often request is the ability to inspect dynamic elements. As an example click inspect, go to a dropdown menu and select a dropdown menu item… when you inspect the item the menu detects the mouse leaving the menu area and the menu collapses. As as a result of the menu collapsing the highlighter highlights where the element was and not where it now is.</p> <p>I am planning on making the inspector modular so that custom highlighters can be added. This will allow extensions to do some cool things like displaying lines when inspecting to show offsets etc.</p> <p>With the transparent overlay we could optionally cancel events sent to the page and send events using an inspector specific context menu instead. The dynamic menu could now be highlighted by right-clicking the parent element, choosing mouseover then inspecting the child node that is now visible.</p> <p>I think that using this technique will make Firebug’s inspector much more powerful… I am also interested in your feedback about the idea would be much appreciated as you may be able to think about things that I have not considered.</p> Sat, 10 Jul 2010 21:34:00 +0100 http://flailingmonkey.com/firebug-inspector-improvements/ http://flailingmonkey.com/firebug-inspector-improvements/ Firebug Lite - Changes Q4 2008 - Q1 2009 Mike Ratcliffe <p>We have made a bunch of changes to <a href="http://getfirebug.com/lite.html">Firebug Lite</a> over the last few months and it is about time we shared them with you. For those that don’t know what Firebug Lite is, it is basically a cross browser version of Firebug. It does not have the JavaScript debugging capabilities of Firebug but is excellent for viewing the HTML and DOM structure of web pages.</p> <p>We have corrected a lot of bugs and added the following enhancements:</p> <ul> <li>Open in Popup - this feature has been requested by lots of people and really improves Firebug Lite:</li> </ul> <figure> <p><img src="/assets/images/firebug-lite-in-popup.jpg" alt="Firebug Lite in Popup" /></p> <figcaption>Firebug lite in popup</figcaption> </figure> <ul> <li> <p>Run the latest development version from a bookmarklet - you think a feature is missing or have found a bug? Have a look at the latest dev version and see if it has been fixed. You can run the latest dev version by using this bookmarklet: <a href="javascript:var%20firebug=document.createElement('script'\);firebug.setAttribute('src','http://fbug.googlecode.com/svn/lite/branches/firebug1.2/firebug-lite.js'\);document.body.appendChild(firebug\);(function(\){if(window.firebug.version\){firebug.init(\);}else{setTimeout(arguments.callee\);}}\)(\);void(firebug\);">Firebug Lite - Dev version</a></p> </li> <li> <p>Firebug Lite icon now appears at the bottom right of the page when Lite is hidden - the icon is a visual indicator that Lite is running. You can click the icon to show Lite, look at the tooltip to see hotkeys or <kbd>ctrl | &#8984;</kbd> + <kbd>click</kbd> to hide the icon.</p> </li> </ul> <figure> <p><img src="/assets/images/firebug-lite-icon.jpg" alt="Firebug Lite Icon" /></p> <figcaption>Firebug lite icon</figcaption> </figure> <ul> <li> <p>No more bleedthrough - there have been numerous issues with Firebug Lite being rendered behind dropdowns, Flash animations, iframes etc. This made it <strong>very</strong> difficult to use for some sites… we now use an iframe shield to prevent this bleedthrough.</p> </li> <li> <p>Added a new preference (firebug.env.detectFirebug) to prevent Lite appearing when Firebug is running - this is now the new default behavior.</p> </li> <li> <p>It is useful when reporting bugs to be able to let us know which version of Firebug Lite you are running. We have added the console.firebug command which displays the full version number.</p> </li> <li>Extra Firebug Lite hotkeys - Lite can be used on a variety of platforms in a variety of browsers. Unfortunately this means that keyboard conflicts are inevitable. To work around this issue we have increased the number of hotkeys that can be used in Lite: <ul> <li>Show Firebug Lite - <kbd>F12</kbd>, <kbd>ctrl | &#8984;</kbd> + <kbd>&#8679;</kbd> + <kbd>L</kbd> or <kbd>&#8679;</kbd> + <kbd>&#9166;</kbd></li> <li>Open in new Window - <kbd>ctrl | &#8984;</kbd> + <kbd>F12</kbd></li> </ul> </li> <li>Options Dropdown - up until now it has been necessary for Lite users to set Lite options in their code in order to control Lite behavior. This is fine if you have full access to a website but it is not possible if you do not have access. To overcome this problem we have added an options dropdown to Firebug Lite. All the old firebug.env settings can still be set in your code but these now just set the default behavior. These settings can now also be changed from the options dropdown and are stored in cookies for each domain.</li> </ul> <figure> <p><img src="/assets/images/firebug-lite-options-dropdown.jpg" alt="Firebug Lite Options Dropdown" /></p> <figcaption>Firebug lite options dropdown</figcaption> </figure> <ul> <li> <p>Text content is no longer trimmed - it was often frustrating that the text content in the HTML tree was trimmed to around 50 characters. This setting is now off by default and can be changed using either firebug.env.textNodeChars or the settings dialog.</p> </li> <li> <p>Hide DOM functions - another recurring issue has been that the DOM tree is slow… especially when using libraries that add a bunch of methods to the DOM. We have added an option to disable the display of DOM functions in the DOM tree. This option can be set using either firebug.env.hideDOMFunctions or the settings dialog.</p> </li> <li> <p>Handling of the same domain policy in XHR, CSS and Script tabs - in the past we always assumed that the same domain policy was active on every site. We have now changed this handling and will always attempt AJAX request (this is how we get the CSS and script text) letting you know only when there is a problem. This means that more scripts and stylesheets will be available for sites that do not use the same domain policy.</p> </li> <li> <p>Text nodes can now be reached via the inspect method. Unfortunately there are limitations to the inspect method in Firebug Lite. This is because of the way that browsers work… as an example imagine that you have a text node inside of a span tag, the span tag contains an onclick handler. If the text is clicked then you probably want the span element to receive the onclick event. For this reason browsers have a tendency to send the onclick event of a text node to the containing element. Because of this issue highlighting or clicking text nodes usually results in highlighting their parent elements. Nevertheless, if a text node is not contained it can now be selected.</p> </li> <li> <p>Lite can now be used locally over HTTPS - unfortunately Lite throws a bunch of insecure content errors when used for HTTPS pages. The only workaround is to copy the files to the domain that you wish to debug and to set firebug.env.css to point to the copy of the css file stored on that domain. Otherwise you just need to click through the warnings when they pop up.</p> </li> <li> <p>We have added these enhancements to Firebug Lite but would like to add more. If you have any ideas of which enhancements we should add or if you find any errors in Lite then please <a href="http://code.google.com/p/fbug/issues/list">log an enhancement request or issue</a>… just don’t forget to mention Firebug Lite in the subject line.</p> </li> </ul> Tue, 05 May 2009 20:54:00 +0100 http://flailingmonkey.com/firebug-lite-changes-q4-2008-q1-2009/ http://flailingmonkey.com/firebug-lite-changes-q4-2008-q1-2009/