tag:support.hglabhq.com,2012-10-18:/discussions/questions/340-web-site-developmentHgLab: Discussion 2015-01-13T00:59:53Ztag:support.hglabhq.com,2012-10-18:Comment/357006952015-01-07T21:05:14Z2015-01-07T21:05:14ZWeb Site Development<div><p>Paul,</p>
<p>This is an interesting question you have. In fact, there are two
entirely separate questions disguised as one:</p>
<ol>
<li>How to version control configuration files (with potentially
sensitive data)<br></li>
<li>How to manage configuration files for different
environments</li>
</ol>
<p>The general consensus for versioning sensitive data is not to
version it at all. For example, <a href=
"http://12factor.net/config">The Twelve-Factor App</a> proposes to
keep all configuration settings in environment variables directly
on the machines your application is running. StackOverflow
questions (<a href="http://stackoverflow.com/q/1974886">1</a>,
<a href="http://stackoverflow.com/q/1396617">2</a> and related)
suggest having a "template" configuration file which is then edited
by each and every developer locally.</p>
<p>As for managing configuration files for different environments,
things depend. If you're using ASP.NET, there's built-in <a href=
"http://msdn.microsoft.com/en-us/library/dd465326(v=vs.110">Web.config
Transformations</a>.aspx), which <em>only</em> works for web
applications. If you're <em>not</em> using ASP.NET but you're still
on .NET, you're effectively out of luck since <a href=
"https://visualstudiogallery.msdn.microsoft.com/69023d00-a4f9-4a34-a6cd-7e854ba318b5">
SlowCheetah</a> (which is a general-purpose XML Transformer) is not
supported anymore.</p>
<p>We've here solved this issue by having a single
<code>.config</code> file which looks like this (note the extra
<code>configuration</code> attribute here and there):</p>
<pre>
<code><connectionStrings>
<add name="hglabhq" connectionString="...initial catalog=hglabhq;integrated security=true" configuration="Debug" />
<add name="hglabhq" connectionString="...initial catalog=hglabhq;user id=yyy;password=yyy" configuration="Beta,QA" />
<add name="hglabhq" connectionString="...initial catalog=hglabhq;user id=%HGLABHQ_USER_ID%;password=%HGLABHQ_PASSWORD%" configuration="Production" />
</connectionStrings>
<system.web>
<trace enabled="true" pageOutput="false" requestLimit="40" localOnly="false" configuration="Debug,Beta,QA" />
</system.web></code>
</pre>
<p>This <code>.config</code> file (which we call
<code>_Web.config</code> or <code>_App.config</code>) is then
transformed (in a Pre-Build Step) into a proper
<code>Web.config</code> or <code>App.config</code> by removing all
the elements with <code>configuration</code> attribute that are not
related to the configuration currently being built. For example,
the "Production" configuration gets its own connection string (that
references environment variables) and does not have the
<code><trace /></code> element at all. This solution is not
ideal, but is <em>far</em> more elegant than having
<code>Web.Debug/Release/QA.config</code> files with wonky transform
syntax.</p>
<p>If you're interested, I can elaborate further on this.</p></div>Anton Gogolevtag:support.hglabhq.com,2012-10-18:Comment/357006952015-01-07T22:20:18Z2015-01-07T22:20:18ZWeb Site Development<div><p>Anton:<br>
Thanks very much for your thoughtful response and the included
links.<br>
Yes, I believe you very succinctly captured the essence of the
question and I agree, it is two fold. The first question was one
that I not sufficiently clued into in my thinking. I have had a
number of discussions with colleagues over the years regarding the
use of config and ini files with clear text in them and oddly
enough no one had ever introduced the idea of env variables. I like
that, it seems to add another layer of obfuscation for the critical
data. Of course, if the server is hacked at a low enough level, the
clever hacker could gain the info, but they'd have to know to look
in the first place. Most of work has been for intranet applications
behind substantial firewalls so many times people concluded that
the firewall is what protects you. That and proper security
settings on files and folders. The basic thought being that a
hacker would have to violate the firewalls first, then find the
internal servers, etc... All of this of course goes out the window
when out in the real world (DMZ.)</p>
<p>I would be grateful if you would elaborate some more on your
Pre-Build step(s).</p>
<p>I'm assuming and guessing at two things: First, that your
web.config files are substantially more complex than seen above.
You're just illustrating the point about using the additional
"configuration" attribute to be able to identify the server for
which this build is intended.</p>
<p>My second guess then is that you have a possibly more elaborate
Pre-Build engine (say a Python script or some C# parser?) that then
strips out all lines that should not be left in the config file for
the intended server build.</p>
<p>When you are stripping the unneeded configuration lines, do you
also strip out the configuration attrib information? Or do you just
leave it in place since it would appear to do no harm?</p>
<p>I'd be grateful for further elaboration and even sample code for
the stripping parser.<br>
I'm a lazy programmer and steal what code I can. :-)</p></div>pdavidsontag:support.hglabhq.com,2012-10-18:Comment/357006952015-01-10T13:37:58Z2015-01-10T13:37:58ZWeb Site Development<div><p>When it comes to storing settings environment variables, a
proper sever automation is key: you really wouldn't want to have
these settings unversioned and manage them manually. Linux world
has <a href="http://puppetlabs.com/">Puppet</a>, <a href=
"https://www.chef.io/chef/">Chef</a>, <a href=
"http://www.ansible.com/home">Ansible</a> and possibly others.
These tools allow you to declaratively specify the state of your
entire sever fleet and then bring all the machines up-to-date.
Windows, unfortunately, has very limited (if any) support from
these three tools, and Microsoft is kind of offering <a href=
"http://blogs.technet.com/b/privatecloud/archive/2013/08/30/introducing-powershell-desired-state-configuration-dsc.aspx">
PowerShell DSC</a>, but as any server-level "enterprise-grade" tool
from Microsoft, it's a disaster.</p>
<p>To solve the above issue, I'm building <a href=
"http://opsflow.io/">Opsflow</a>.</p>
<p>Now to the configuration file transformation. It's actually very
simple. First, you'll need a trivial XSLT file:</p>
<pre>
<code><?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" />
<xsl:param name="Configuration" />
<xsl:template match="*">
<xsl:if test="not(@configuration) or contains(@configuration, $Configuration)">
<xsl:copy>
<xsl:for-each select="@*[name() != 'configuration']">
<xsl:copy />
</xsl:for-each>
<xsl:apply-templates />
</xsl:copy>
</xsl:if>
</xsl:template>
</xsl:stylesheet></code>
</pre>
<p>What it does is it loops over each and every tag in an XML; if
the tag does not have a <code>configuration</code> attribute, just
copies it over to "output"; if it does, it checks if this attribute
contains a string passed into this XSLT file from the outside via
<code>Configuration</code> parameter and either copies the tag
(sans <code>configuration</code>) attribute, or leaves it out.</p>
<p>Now, you'll need a command-line tool to apply XSLT
transformation to an arbitrary XML file. I use
<code>nxslt2.exe</code>.</p>
<p>Finally, just glue these pieces together in a Pre-Build
step:</p>
<pre>
<code>"$(ProjectDir)..\..\tools\nxslt2.exe" "$(ProjectDir)_Web.config" "$(ProjectDir)..\..\tools\configuration-transformer.xslt" -o "$(ProjectDir)Web.config" Configuration="$(ConfigurationName)"</code>
</pre>
<p>(Note that both <code>nxslt2.exe</code> and
<code>configuration-transformer.xslt</code> are kept inside the
repository in a top-level <code>/tools</code> directory).</p>
<p>This command basically says: "Hey, nxslt2! Transform
<code>$(ProjectDir)_Web.config</code> using
<code>$(ProjectDir)..\..\tools\configuration-transformer.xslt</code>
as a transformation passing it the <code>Configuration</code>
parameter and write output to
<code>$(ProjectDir)Web.config</code>".</p>
<p>That's how <code>_Web.config</code> with those "magical"
<code>configuration</code> attributes gets transformed into proper
.NET-compatible <code>Web.config</code>. In fact, you can use this
approach with any XML and any XSLT file -- be it
<code>App.config</code> or anything else, really.</p></div>Anton Gogolevtag:support.hglabhq.com,2012-10-18:Comment/357006952015-01-13T00:59:51Z2015-01-13T00:59:51ZWeb Site Development<div><p>Anton:</p>
<p>Thank you very much for this information.<br>
It's very helpful.</p>
<p>I read through the 12-Factor website and while their concepts
make a lot of sense as best practices, it's, as you point out, not
so simply done in the Windows world.</p>
<p>Look forward to your Opsflow package. I believe we would use a
package like that almost immediately.</p>
<p>I have looked at Vagrant and it appears to probably be a useful
tool once we get past some initial issues (below.)<br>
As best I can tell from only reading, Vagrant gives you the ability
to create Virtual environments so that you can then have everyone
working under the same environment. This would certainly help with
some of the issues we're fighting with given our current
systems.</p>
<p>The main issue we are currently struggling with is that there
was not a test server in place within our organization. We're
implementing one but in doing so finding out that the development
and production environments were not kept in synch. Not even at the
Framework level. It appears the source was copied from one place to
another and then recompiled to match the environment. You can
imagine the issues this can lead to...</p>
<p>Your information, examples and help are very useful for us right
now.<br>
We should be able to implement them to help get things cleaned
up.</p></div>pdavidson