How to Upload Large Files

4 min read Uploading files these days is pretty straightforward in just about any web framework and programming language, however, when files get big or many files are uploaded at the same time, memory usage starts being a concern and starts to run into bottlenecks. Aside from this, frameworks put in constraints to protect the application from things like denial of service through resource exhaustion. I ran into several of these limitations over the years and came up with a few solutions. The examples below are using the MVC framework in .net core…

by Alexandru Puiu
December 24
in
- C# / .Net

5 min read Foreach itself is very useful and efficient for most operations. Sometimes special situations arise where high latency in getting data to iterate over, or processing data inside the foreach depends on an operation with very high latency or long processing. This is the case for example with getting paged data from a database to iterate over. The goal is to start getting data from the database, but a chunk of data at a time, since getting one record at a time introduces its own overhead. As the data becomes available, we’d start processing it, while in the background we get more data and feed it into the processor. The processing part would itself be parallel as well, and start processing the next iterator.

ForEachAsync

My favorite way to do this is with an extension method Stephen Toub wrote many years ago, that accepts a data generator and breaks the data source into partitions allowing for specifying the degree of parallelism and accepts a lambda to execute for each item

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace Extensions
{
    public static class Extensions
    {
        public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
        {
            return Task.WhenAll(
                from partition in Partitioner.Create(source).GetPartitions(dop)
                select Task.Run(async delegate
                {
                    using (partition)
                        while (partition.MoveNext())
                            await body(partition.Current);
                }));
        }
    }
}

But let’s see what we can do to optimize more…

by Alexandru Puiu
February 8
in
- C# / .Net

10 min read

Building the graph

The first challenge we have is representing the capabilities of the system in a logical way. We can get a list of all the possible general classes of states, and represent inheritance for them, which will come in handy later.

var derivedTypes = ReflectionHelpers.FindAllDerivedTypes<ProcessableState>();
foreach (var type in derivedTypes)
    graphClient.Cypher
        .Create("(statetype:StateType {newState})")
        .WithParam("newState", new { type.Name })
        .ExecuteWithoutResults();

this is done with a simple extension class

public class ReflectionHelpers
{
    public static List<Type> FindAllDerivedTypes<T>()
    {
        return FindAllDerivedTypes<T>(Assembly.GetAssembly(typeof(T)));

…

by Alexandru Puiu
October 30
in
- C# / .Net

2 min read

What is the function of HSTS

HSTS stands for HTTP Strict Transport Security and it tells your browser that your web content should always be served over HTTPS. See Security Headers for more info

Adding a signed localhost certificate to the Trusted Root Certification Authorities store

Newer versions of chrome require the server’s cert must contain a “subjectAltName” otherwise known as a SAN certificate. If you are using an older signed certificate which only references a commonName, then you might still get rejected by Chrome even if you’re certificate is valid.

Large sitemap

3 min read We built a social network-type site recently with quite a large number of public-facing pages, and like any good site, we created a sitemap with links to every page. All fine on launch, every page we tested finished in under 3ms of logic processing, and everyone was very happy. A few weeks pass and everything looks great, until all of a sudden all our request times go through the roof, and the database is maxing out and we’re scratching our heads at what’s causing all the load. No abnormal requests,…

by Alexandru Puiu
November 3
in
- C# / .Net
- MVC 3

< 1 min read

public IEnumerable Months
{
    get
    {
        return DateTimeFormatInfo
               .InvariantInfo
               .MonthNames
               .Select((monthName, index) =&gt; new SelectListItem
               {
                   Value = (index + 1).ToString(),
                   Text = monthName
               });
    }
}

by Alexandru Puiu
October 8
in
- C# / .Net

< 1 min read Quick script for generating an MD5 Hash

private string ComputeHash(string input)
{
  using (var md5 = System.Security.Cryptography.MD5.Create())
  {
    var data = md5.ComputeHash(Encoding.UTF8.GetBytes(input));
    var sb = new StringBuilder();
    foreach (var c in data) {
        sb.Append(c.ToString("x2"));
    }
    return sb.ToString();
  }
}

by Alexandru Puiu
October 8
in
- MVC 3
- WCF

2 min read As you might’ve noticed, keeping threads running after a request returns, for processing post operational tasks (such as performing analytics on a file that was uploaded, etc) don’t always complete in a web project.
There are several issues with spawning threads in the context of an ASP.NET project. Phil Haack’s post explains the issues in more detail. The following classes solve the problem of IIS killing threads before they complete.

Value types, reference types and practical uses

2 min read C# has two different types of variables: value types and reference types. While in C and C++ primitive types can contain values or references and certain complex types (arrays, objects) can only be used via reference, in C# the line between the two types is very clear. Numeric types(int, decimal, double, etc.), bool and structs access the values directly. Class, object, interface, delegate string and dynamic are only accessed and used via reference. Because of all the awkwardness with referencing and dereferencing in C and C++, C# uses the ‘ref’ keyword only for those cases where you want to modify a value type outside of your current scope.

If you’ve worked with C# a bit you’re probably used to always working with copies in any situation outside of simple assignment (=). If you loop over a collection, inside the loop you’re working with copies. If you pass something to a function, that something is a copy. However the difference between value types and reference types often means a copy isn’t always a copy. Knowing when this is not the case can help you spot troublesome bugs, and easily solve problems that might have taken you much longer.

by Alexandru Puiu
April 26
in
- LINQ

< 1 min read Updated with statements that can be executed or deferred.

// Example models
public class SomeModel
{
    public int Id { get; set; }
    public string Name { get; set; }
    public List Items { get; set; }
}
public class SomeOtherModel
{
    public int Id { get; set; }
    public string Text { get; set; }
}
//Expression mapper - not executed
public Expr<Func> Map_SomeTable_SomeModel_Expr()
{
    return x => new SomeModel
    {
        Id = x.Id,
        Name = x.Name,
        Items = db.SomeOtherTable.Where(y => y.SomeTableId == x.Id).Select(Map_SomeOtherTable_SomeOtherModel_Expr()).ToList()
    }
}

How to Upload Large Files

Parallel Foreach async in C#

ForEachAsync

Representing State Machine Capabilities using neo4j / graph databases

Building the graph

Debugging on localhost with HSTS

What is the function of HSTS

Adding a signed localhost certificate to the Trusted Root Certification Authorities store

Large sitemap

C# MVC Drop down List for Months

C# MD5 Hash

Background tasks in MVC and IIS

Value types, reference types and practical uses

Mapping database objects to models – the Lambda way

Tags

Categories

Popular Posts

ForEachAsync

Building the graph

What is the function of HSTS

Adding a signed localhost certificate to the Trusted Root Certification Authorities store

Social Media

Tags

Categories

Popular Posts