Home → Questions →

Web Site Development

• Edit

pdavidson

06 Jan, 2015 02:16 AM

Anton (and any others willing to contribute):

This is more a question about DVCS than HgLabs but I've tried to do some research on the net and haven't found anything all that on point.

Is there a best practice for using DVCS in a web development environment?

For example, assume one has a local dev box, a test box in the network and then a production box on the network (either internal or external.)

Often when doing web work, just after a new release, all three of these setups may contain code that is about 99% the same.
The only difference often will just be the web.config file because it will often point at different data sources, error pages, etc...

How do most folks handle this?
Do you exclude the web.config file from the repository?
  - This is not an option I like since for me, a huge advantage to source control is the use as a code backup system and this excludes web.config from getting backed up. However, I've found that over time, the web.config files on the different boxes diverges more and more.

Do you create a sub folder in the system that just holds copies of the three web.config files. Say, web.config.LocalDev, web.config.test, web.config.prod. Then you modify these as you need to and then when testing, copy the appropriate file into the web sites root folder?
  - This at least has the advantage of backing everything up.

Or do you Fork a seperate version for each of the three websites?
 - This seems overkill and probably not at all what I want.

Comments? Suggestions?

1. Support Staff 1 Posted by Anton Gogolev on 07 Jan, 2015 09:05 PM

   

   Paul,

   This is an interesting question you have. In fact, there are two entirely separate questions disguised as one:

   1. How to version control configuration files (with potentially sensitive data)
      
   2. How to manage configuration files for different environments

   The general consensus for versioning sensitive data is not to version it at all. For example, The Twelve-Factor App proposes to keep all configuration settings in environment variables directly on the machines your application is running. StackOverflow questions (1, 2 and related) suggest having a "template" configuration file which is then edited by each and every developer locally.

   As for managing configuration files for different environments, things depend. If you're using ASP.NET, there's built-in Web.config Transformations.aspx), which only works for web applications. If you're not using ASP.NET but you're still on .NET, you're effectively out of luck since SlowCheetah (which is a general-purpose XML Transformer) is not supported anymore.

   We've here solved this issue by having a single .config file which looks like this (note the extra configuration attribute here and there):

   <connectionStrings>
     <add name="hglabhq" connectionString="...initial catalog=hglabhq;integrated security=true" configuration="Debug" />
     <add name="hglabhq" connectionString="...initial catalog=hglabhq;user id=yyy;password=yyy" configuration="Beta,QA" />
     <add name="hglabhq" connectionString="...initial catalog=hglabhq;user id=%HGLABHQ_USER_ID%;password=%HGLABHQ_PASSWORD%" configuration="Production" />
   </connectionStrings>
   
   <system.web>
     <trace enabled="true" pageOutput="false" requestLimit="40" localOnly="false" configuration="Debug,Beta,QA" />
   </system.web>
   

   This .config file (which we call _Web.config or _App.config) is then transformed (in a Pre-Build Step) into a proper Web.config or App.config by removing all the elements with configuration attribute that are not related to the configuration currently being built. For example, the "Production" configuration gets its own connection string (that references environment variables) and does not have the <trace /> element at all. This solution is not ideal, but is far more elegant than having Web.Debug/Release/QA.config files with wonky transform syntax.

   If you're interested, I can elaborate further on this.

   • Edit

2. 2 Posted by pdavidson on 07 Jan, 2015 10:20 PM

   

   Anton:
   Thanks very much for your thoughtful response and the included links.
   Yes, I believe you very succinctly captured the essence of the question and I agree, it is two fold. The first question was one that I not sufficiently clued into in my thinking. I have had a number of discussions with colleagues over the years regarding the use of config and ini files with clear text in them and oddly enough no one had ever introduced the idea of env variables. I like that, it seems to add another layer of obfuscation for the critical data. Of course, if the server is hacked at a low enough level, the clever hacker could gain the info, but they'd have to know to look in the first place. Most of work has been for intranet applications behind substantial firewalls so many times people concluded that the firewall is what protects you. That and proper security settings on files and folders. The basic thought being that a hacker would have to violate the firewalls first, then find the internal servers, etc... All of this of course goes out the window when out in the real world (DMZ.)

   I would be grateful if you would elaborate some more on your Pre-Build step(s).

   I'm assuming and guessing at two things: First, that your web.config files are substantially more complex than seen above. You're just illustrating the point about using the additional "configuration" attribute to be able to identify the server for which this build is intended.

   My second guess then is that you have a possibly more elaborate Pre-Build engine (say a Python script or some C# parser?) that then strips out all lines that should not be left in the config file for the intended server build.

   When you are stripping the unneeded configuration lines, do you also strip out the configuration attrib information? Or do you just leave it in place since it would appear to do no harm?

   I'd be grateful for further elaboration and even sample code for the stripping parser.
   I'm a lazy programmer and steal what code I can. :-)

   • Edit

3. Support Staff 3 Posted by Anton Gogolev on 10 Jan, 2015 01:37 PM

   

   When it comes to storing settings environment variables, a proper sever automation is key: you really wouldn't want to have these settings unversioned and manage them manually. Linux world has Puppet, Chef, Ansible and possibly others. These tools allow you to declaratively specify the state of your entire sever fleet and then bring all the machines up-to-date. Windows, unfortunately, has very limited (if any) support from these three tools, and Microsoft is kind of offering PowerShell DSC, but as any server-level "enterprise-grade" tool from Microsoft, it's a disaster.

   To solve the above issue, I'm building Opsflow.

   Now to the configuration file transformation. It's actually very simple. First, you'll need a trivial XSLT file:

   <?xml version="1.0" encoding="UTF-8" ?>
   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
     <xsl:output method="xml" />
   
     <xsl:param name="Configuration" />
   
     <xsl:template match="*">    
       <xsl:if test="not(@configuration) or contains(@configuration, $Configuration)">
         <xsl:copy>      
           <xsl:for-each select="@*[name() != 'configuration']">          
             <xsl:copy />
           </xsl:for-each>
   
           <xsl:apply-templates />
         </xsl:copy>
       </xsl:if>
     </xsl:template>
   </xsl:stylesheet>
   

   What it does is it loops over each and every tag in an XML; if the tag does not have a configuration attribute, just copies it over to "output"; if it does, it checks if this attribute contains a string passed into this XSLT file from the outside via Configuration parameter and either copies the tag (sans configuration) attribute, or leaves it out.

   Now, you'll need a command-line tool to apply XSLT transformation to an arbitrary XML file. I use nxslt2.exe.

   Finally, just glue these pieces together in a Pre-Build step:

   "$(ProjectDir)..\..\tools\nxslt2.exe" "$(ProjectDir)_Web.config" "$(ProjectDir)..\..\tools\configuration-transformer.xslt" -o "$(ProjectDir)Web.config" Configuration="$(ConfigurationName)"
   

   (Note that both nxslt2.exe and configuration-transformer.xslt are kept inside the repository in a top-level /tools directory).

   This command basically says: "Hey, nxslt2! Transform $(ProjectDir)_Web.config using $(ProjectDir)..\..\tools\configuration-transformer.xslt as a transformation passing it the Configuration parameter and write output to $(ProjectDir)Web.config".

   That's how _Web.config with those "magical" configuration attributes gets transformed into proper .NET-compatible Web.config. In fact, you can use this approach with any XML and any XSLT file -- be it App.config or anything else, really.

   • Edit

4. 4 Posted by pdavidson on 13 Jan, 2015 12:59 AM

   

   Anton:

   Thank you very much for this information.
   It's very helpful.

   I read through the 12-Factor website and while their concepts make a lot of sense as best practices, it's, as you point out, not so simply done in the Windows world.

   Look forward to your Opsflow package. I believe we would use a package like that almost immediately.

   I have looked at Vagrant and it appears to probably be a useful tool once we get past some initial issues (below.)
   As best I can tell from only reading, Vagrant gives you the ability to create Virtual environments so that you can then have everyone working under the same environment. This would certainly help with some of the issues we're fighting with given our current systems.

   The main issue we are currently struggling with is that there was not a test server in place within our organization. We're implementing one but in doing so finding out that the development and production environments were not kept in synch. Not even at the Framework level. It appears the source was copied from one place to another and then recompiled to match the environment. You can imagine the issues this can lead to...

   Your information, examples and help are very useful for us right now.
   We should be able to implement them to help get things cleaned up.

   • Edit