Requirements of the Project:
Privly has several layers of continuous integration testing needs:
1. Privly Applications: Web apps that may run inside many different contexts.
Contexts:
- Browser extension injected
- Browser extension as the top window
- Hosted (by a content server)
- Mobile apps
2. Browser Extensions: Here we want to be able to run several combinations of browsers and versions with multiple versions of the extensions and the privly-applications bundle. Tests for the browser extensions are different from the Privly Applications in that they are in the "chrome"--meaning they are part of the API exposed by the browser extension framework.
3. Host Page Integration Tests: We want to be able to tell when test links fail to properly inject into the page. This involves scripting web browsers to detect whether links have been injected on particular pages.
4. Content Server
Listed in order of importance, testing Privly Applications in different contexts is the priority.
Developers will likely make these "Privly Applications" on one platform and want compatibility reports on all the other platforms since they are unlikely to take the time to manually test for every browser and browser version.
Privly Applications currently have tests written with Jasmine, which is a JavaScript library for expressing and running tests.
Problems that need to be overcome:
1. Testing privly-applications requires automated testing. i.e. Setting up the entire Privly Foundation System (Content Server + Privly Applications) and then loading privly-applications into real web browsers and running test on them and then pass reports on the tests to Travis so that we can get the build status (pass/fail).
CIS Structure:
Approach:
Integrating with Travis:
For the Continuous Integration System first of all we need Integration with any Hosted Continuous Integration Platform for Open Source Community. The best option being Travis CI.
Thus firstly the Content Server (privly-web) was integrated with Travis and the content server tests where run on it and its reports were logged.
https://groups.google.com/forum/#!topic/privly/iimgyei_Sb8
Adding JavaScript tests to CIS:
https://github.com/privly/privly-applications/issues/35
Existing implementation of JavaScript testing:
After going through the implementation and documentation some main points:
1. For consistency across all the JavaScript projects, Privly is using the Jasmine testing library for running tests.
2. Use of console reporter which sends test results directly to the JavaScript console.
3. And also the use of meta_loader.js which finds the PrivlySpec meta tag and uses it to inject the appropriate testing scripts and specs.
This implementation allows to test the privly-applications by running the runTests() command in the JavaScript console which loads the Spec files and runs the tests. It then shows the output in the console itself.
Problem: For this to be integrated with Travis we need to print this output to the terminal.
Solution: Repo
This repo gives details about how to run jasmine tests with phantomjs and selenium-webdriver and get the results to the terminal.
After discussing we finally decided to go with selenium-webdriver for running tests.
Reasons: Link #2 and #5 being the main reasons.
Getting test results to the terminal:
This implementation requires the Sources files and the Spec files to be present in the HTML itself.
Thus the system requires to make changes to the structure of privly-applications that currently uses meta_loader.js to load Spec files from the meta tags.
Privly-Applications repo has different applications in it. Like ZeroBin, PlainPost etc.
All the apps have new.html and a show.html files that do app specific tasks.
These new.html and show.html files are generated by a script build.py that takes a template file new.html.template and show.html.template from the templates directory and new.html.subtemplate and show.html.subtemplate from the app directory to finally build the new.html and show.html file.
So change the structure of privly-applications I had to makes changes to the .template and .subtemplate files so that they don't use the meta_loader.js to load the spec files from the <meta> tags. The new structure instead requires all the app specific specs to be already included in the HTML as JavaScript files.
So I made changes to the .template file wherein I removed the use of meta_loader.js and added shared JavaScript files that are common to all the privly-applications. The jasmine library and the console-runner are also included in the .template file only. Then I made changes to specific .subtemplate files of each app wherein I added app specific spec files.
Then I ran the `build.py` script (make sure all the dependencies are installed) which generated the new.html and show.html files for all the apps depending upon the new .template and .subtemplate files.
This implementation gives us two main methods:
1. consoleReporter.status (tests passed or not)
2. consoleReporter.getLogsAsString() (logs the reports of all tests that were run)
Now we can use these methods to log the reports to the terminal.
Using Selenium-Webdriver for Automated testing:
Now that we have the methods that we can use to log the reports to the terminal. Now comes up the task to load all privly-application web pages in real browsers and run tests over them and log the reports. For this we will use selenium webdriver.
Sticking to the ruby way, we'll be using selenium-webdriver gem and rake tasks to perform the testing operations and the rake tasks can be easily integrated with Travis too.
Reference code: https://github.com/jphpsf/jasmine-phantomjs/blob/master/Rakefile
Selenium-Webdriver documentation: https://code.google.com/p/selenium/wiki/RubyBindings
In the reference code they use local chromium and firefox browsers, but in our case we could only use firefox as it pre-installed in the Travis VM.
Then I added all the privly-application pages to be tested to the test_pages array and then using the driver options loaded them one by one and ran the scripts to run the tests and log the reports.
After being able to run tests on all privly-applications, now comes the task to run these tests on different browsers and versions on different platforms.
Integration with SauceLabs:
To test the privly-applications on different browsers and versions on different platforms we need to integrate Travis and Selenium-webdriver with SauceLabs.
Sauce Labs' cloud testing platform allows to automatically or interactively test mobile and web applications on 300+ browsers and platforms.
http://docs.travis-ci.com/user/sauce-connect/
https://docs.saucelabs.com/tutorials/ruby/
https://saucelabs.com/docs/ondemand/getting-started/env/ruby/se2/windows
Following these links I setup the integration of Travis + Selenium Webdriver + SauceLabs and now was able to run tests on different browsers, browser versions and on different platforms.
Testing privly-applications in Extension context:
My privly-applications branch was now different to the master privly-applications branch of Privly Foundation. I had to clone privly-chrome and privly-firefox repos and then replace my version of privly-applications in it to start the process of testing privly-applications in extension context.
After replacing I manually did the build extension process for both privly-chrome and privly-firefox
https://developer.chrome.com/extensions/getstarted#unpacked
https://developer.mozilla.org/en-US/docs/Building_an_Extension#Package
For proceeding towards figuring out extension testing, I for the time being copied the updated .crx file and the .xpi file to my privly-web repo, so that I can load it to the remote driver and run privly-application tests in extension context.
NOTE: Help in terms of Documentation and Answers over StackOverflow related to extension testing is available for Java and very less in case of Ruby.
Now the first step was to add the extension to the remote driver.
https://code.google.com/p/selenium/wiki/RubyBindings#Adding_an_extension
And now we need to load the testing URLs in the extension context. e.g.
chrome://privly/content/privly-applications/Help/new.html
and then follow the same process of running the tests. But, in this case I got several errors. Firstly, it takes time for the extension to load in the browser, the URLs where loading before the extension could properly load hence the tests weren't running properly. Then came the Timeout::Error. On goggling a lot, it was found that it depends on driver manage timeouts.
driver.manage.timeouts.implicit_wait = 300
driver.manage.timeouts.script_timeout = 300
driver.manage.timeouts.page_load = 10
Discussion related to this is going on with Sauce Labs' guys.
http://support.saucelabs.com/requests/13977
Users Guide:
Final code repos:
https://github.com/dreamrulez07/privly-web/tree/privlyCIS
https://github.com/dreamrulez07/privly-applications/tree/privlyCIS
Clone both the branches. Move all the files from the privly-applications repo to public/apps in privly-web.
Install the development server locally:
https://github.com/dreamrulez07/privly-web/tree/privlyCIS#development-server-installation
Start the rails server.
To test privly-applications in Chrome and Firefox browser, run the command:
rake webdriver test=localappsWeb
To test privly-applications in Firefox Extension context:
Firstly, clone the privly-firefox repo
https://github.com/privly/privly-firefox
Copy my privly-applications version to /chrome/content. Zip the folder and rename it as .xpi
Include the .xpi in the privly-web folder. Then run the command:
rake webdriver test=localappsExten
For Automated testing, similar rake tasks have been written with Remote drivers that run the tests on Sauce Labs and report the results to Travis
Commands:
rake webdriver test=appsWeb
rake webdriver test=appsExten
Include these tasks in the .travis.yml file under the script section:
- RAILS_ENV=test bundle exec rake --trace webdriver test=appsWeb
Areas that still need to be addressed:
1. Testing privly-applications in Chrome extension context. Chrome uses something called Chromedriver to run tests, which is not properly integrated with selenium-webdriver gem.
2. Travis at times gives Timeout::Error, need to resolve those errors. Discussion going on with Sauce Labs people related to the same.
3. Need to plan out a way in which the prilvy-firefox and privly-chrome extensions create a new build automatically when new code is committed to the privly-applications repo and then the tests run over them.
Privly has several layers of continuous integration testing needs:
1. Privly Applications: Web apps that may run inside many different contexts.
Contexts:
- Browser extension injected
- Browser extension as the top window
- Hosted (by a content server)
- Mobile apps
2. Browser Extensions: Here we want to be able to run several combinations of browsers and versions with multiple versions of the extensions and the privly-applications bundle. Tests for the browser extensions are different from the Privly Applications in that they are in the "chrome"--meaning they are part of the API exposed by the browser extension framework.
3. Host Page Integration Tests: We want to be able to tell when test links fail to properly inject into the page. This involves scripting web browsers to detect whether links have been injected on particular pages.
4. Content Server
Listed in order of importance, testing Privly Applications in different contexts is the priority.
Developers will likely make these "Privly Applications" on one platform and want compatibility reports on all the other platforms since they are unlikely to take the time to manually test for every browser and browser version.
Privly Applications currently have tests written with Jasmine, which is a JavaScript library for expressing and running tests.
Problems that need to be overcome:
1. Testing privly-applications requires automated testing. i.e. Setting up the entire Privly Foundation System (Content Server + Privly Applications) and then loading privly-applications into real web browsers and running test on them and then pass reports on the tests to Travis so that we can get the build status (pass/fail).
- Current system logs the test results in the JavaScript console, we need to get the results to the terminal to check if the build in passing or not.
- We need to automate the process of loading each privly-application page and running tests on it.
CIS Structure:
Approach:
Integrating with Travis:
For the Continuous Integration System first of all we need Integration with any Hosted Continuous Integration Platform for Open Source Community. The best option being Travis CI.
Thus firstly the Content Server (privly-web) was integrated with Travis and the content server tests where run on it and its reports were logged.
https://groups.google.com/forum/#!topic/privly/iimgyei_Sb8
Adding JavaScript tests to CIS:
https://github.com/privly/privly-applications/issues/35
Existing implementation of JavaScript testing:
After going through the implementation and documentation some main points:
1. For consistency across all the JavaScript projects, Privly is using the Jasmine testing library for running tests.
2. Use of console reporter which sends test results directly to the JavaScript console.
3. And also the use of meta_loader.js which finds the PrivlySpec meta tag and uses it to inject the appropriate testing scripts and specs.
This implementation allows to test the privly-applications by running the runTests() command in the JavaScript console which loads the Spec files and runs the tests. It then shows the output in the console itself.
Problem: For this to be integrated with Travis we need to print this output to the terminal.
Solution: Repo
This repo gives details about how to run jasmine tests with phantomjs and selenium-webdriver and get the results to the terminal.
After discussing we finally decided to go with selenium-webdriver for running tests.
Reasons: Link #2 and #5 being the main reasons.
Getting test results to the terminal:
This implementation requires the Sources files and the Spec files to be present in the HTML itself.
Thus the system requires to make changes to the structure of privly-applications that currently uses meta_loader.js to load Spec files from the meta tags.
Privly-Applications repo has different applications in it. Like ZeroBin, PlainPost etc.
All the apps have new.html and a show.html files that do app specific tasks.
These new.html and show.html files are generated by a script build.py that takes a template file new.html.template and show.html.template from the templates directory and new.html.subtemplate and show.html.subtemplate from the app directory to finally build the new.html and show.html file.
So change the structure of privly-applications I had to makes changes to the .template and .subtemplate files so that they don't use the meta_loader.js to load the spec files from the <meta> tags. The new structure instead requires all the app specific specs to be already included in the HTML as JavaScript files.
So I made changes to the .template file wherein I removed the use of meta_loader.js and added shared JavaScript files that are common to all the privly-applications. The jasmine library and the console-runner are also included in the .template file only. Then I made changes to specific .subtemplate files of each app wherein I added app specific spec files.
Then I ran the `build.py` script (make sure all the dependencies are installed) which generated the new.html and show.html files for all the apps depending upon the new .template and .subtemplate files.
This implementation gives us two main methods:
1. consoleReporter.status (tests passed or not)
2. consoleReporter.getLogsAsString() (logs the reports of all tests that were run)
Now we can use these methods to log the reports to the terminal.
Using Selenium-Webdriver for Automated testing:
Now that we have the methods that we can use to log the reports to the terminal. Now comes up the task to load all privly-application web pages in real browsers and run tests over them and log the reports. For this we will use selenium webdriver.
Sticking to the ruby way, we'll be using selenium-webdriver gem and rake tasks to perform the testing operations and the rake tasks can be easily integrated with Travis too.
Reference code: https://github.com/jphpsf/jasmine-phantomjs/blob/master/Rakefile
Selenium-Webdriver documentation: https://code.google.com/p/selenium/wiki/RubyBindings
In the reference code they use local chromium and firefox browsers, but in our case we could only use firefox as it pre-installed in the Travis VM.
Then I added all the privly-application pages to be tested to the test_pages array and then using the driver options loaded them one by one and ran the scripts to run the tests and log the reports.
After being able to run tests on all privly-applications, now comes the task to run these tests on different browsers and versions on different platforms.
Integration with SauceLabs:
To test the privly-applications on different browsers and versions on different platforms we need to integrate Travis and Selenium-webdriver with SauceLabs.
Sauce Labs' cloud testing platform allows to automatically or interactively test mobile and web applications on 300+ browsers and platforms.
http://docs.travis-ci.com/user/sauce-connect/
https://docs.saucelabs.com/tutorials/ruby/
https://saucelabs.com/docs/ondemand/getting-started/env/ruby/se2/windows
Following these links I setup the integration of Travis + Selenium Webdriver + SauceLabs and now was able to run tests on different browsers, browser versions and on different platforms.
Testing privly-applications in Extension context:
My privly-applications branch was now different to the master privly-applications branch of Privly Foundation. I had to clone privly-chrome and privly-firefox repos and then replace my version of privly-applications in it to start the process of testing privly-applications in extension context.
After replacing I manually did the build extension process for both privly-chrome and privly-firefox
https://developer.chrome.com/extensions/getstarted#unpacked
https://developer.mozilla.org/en-US/docs/Building_an_Extension#Package
For proceeding towards figuring out extension testing, I for the time being copied the updated .crx file and the .xpi file to my privly-web repo, so that I can load it to the remote driver and run privly-application tests in extension context.
NOTE: Help in terms of Documentation and Answers over StackOverflow related to extension testing is available for Java and very less in case of Ruby.
Now the first step was to add the extension to the remote driver.
https://code.google.com/p/selenium/wiki/RubyBindings#Adding_an_extension
And now we need to load the testing URLs in the extension context. e.g.
chrome://privly/content/privly-applications/Help/new.html
and then follow the same process of running the tests. But, in this case I got several errors. Firstly, it takes time for the extension to load in the browser, the URLs where loading before the extension could properly load hence the tests weren't running properly. Then came the Timeout::Error. On goggling a lot, it was found that it depends on driver manage timeouts.
driver.manage.timeouts.implicit_wait = 300
driver.manage.timeouts.script_timeout = 300
driver.manage.timeouts.page_load = 10
Discussion related to this is going on with Sauce Labs' guys.
http://support.saucelabs.com/requests/13977
Final code repos:
https://github.com/dreamrulez07/privly-web/tree/privlyCIS
https://github.com/dreamrulez07/privly-applications/tree/privlyCIS
Clone both the branches. Move all the files from the privly-applications repo to public/apps in privly-web.
Install the development server locally:
https://github.com/dreamrulez07/privly-web/tree/privlyCIS#development-server-installation
Start the rails server.
To test privly-applications in Chrome and Firefox browser, run the command:
rake webdriver test=localappsWeb
To test privly-applications in Firefox Extension context:
Firstly, clone the privly-firefox repo
https://github.com/privly/privly-firefox
Copy my privly-applications version to /chrome/content. Zip the folder and rename it as .xpi
Include the .xpi in the privly-web folder. Then run the command:
rake webdriver test=localappsExten
For Automated testing, similar rake tasks have been written with Remote drivers that run the tests on Sauce Labs and report the results to Travis
Commands:
rake webdriver test=appsWeb
rake webdriver test=appsExten
Include these tasks in the .travis.yml file under the script section:
- RAILS_ENV=test bundle exec rake --trace webdriver test=appsWeb
- RAILS_ENV=test bundle exec rake --trace webdriver test=appsExten
P.S. make sure chromedriver is installed locally to test privly-applications using Chrome browser:
Areas that still need to be addressed:
1. Testing privly-applications in Chrome extension context. Chrome uses something called Chromedriver to run tests, which is not properly integrated with selenium-webdriver gem.
2. Travis at times gives Timeout::Error, need to resolve those errors. Discussion going on with Sauce Labs people related to the same.
3. Need to plan out a way in which the prilvy-firefox and privly-chrome extensions create a new build automatically when new code is committed to the privly-applications repo and then the tests run over them.
